If you cannot replay Tuesday, you do not have observability

A team showed us their Grafana. Twenty-seven panels. CPU looking serene. p95 “a bit spiky”. The customer had still waited 40 seconds to place an order on Tuesday and nobody could say which hop did it. That is not observability. That is interior design.

We use a blunt test. Can an engineer who was not on call reconstruct one failed user action from yesterday without SSH and folklore? If the answer is a slide about “maturity”, you are not ready for the traffic you are courting.

Maturity models are how vendors sell the next SKU. Replay is how you sleep. We will take a messy OpenTelemetry setup that can tell the story of one checkout over a beautiful red/green wall that cannot.

Three artefacts, not thirty dashboards

We ask for a request id that follows the user through the browser, the API, the queue, and the worker. We ask for structured logs with that id, the tenant, and the decision the code took — not a sentence a human thought was clever at 1 a.m. We ask for traces on the paths that touch money, login, or anything a regulator might request.

Metrics still matter. They tell you when to wake someone. They do not tell you what happened to Mrs. Iyer’s payment. For that you need the story of one request.

Put the request id on the error page the user sees, or in the support console next to the ticket. If support has to ask engineering for a needle in CloudWatch, you built a museum, not a system. We have watched 15-minute incidents become 90-minute ones for that reason alone.

What a useful log line looks like

Not this: `Error processing request`. That line has been in production since before some of your engineers could vote. It will still be there in 2028 if you let it.

This: `payout.rejected` with `request_id`, `tenant_id`, `payout_id`, `reason=insufficient_balance`, `provider=razorpay`, `latency_ms=214`. Now a human can act. So can a dashboard. So can an alert that is not a regex on English.

• JSON. Always. Your future self will not regex a novel.
• One event per decision. “auth.denied”, “payout.queued”, “invoice.render.failed”.
• Never log secrets, tokens, or full PAN / Aadhaar / health payloads. Hash or last-four if you must correlate.
• A sample of successful requests, not only errors. Outages hide in the shape of “success”.
• A standard field list in the template repo so every new service does not invent `userId` versus `user_id`.

OpenTelemetry is the boring choice. Take it.

Pick a vendor later. Datadog, Grafana Cloud, New Relic, Honeycomb — we have shipped on all of them. The portable part is the instrumentation. If your traces only exist as a proprietary agent sprinkled through a few services, you will rewrite this in 18 months when procurement changes their mind.

Instrument the edges first: incoming HTTP, outgoing HTTP, SQL, and the queue. That covers most of the “where did the time go” conversations. Fancy custom spans can wait until you have a question they would answer.

Sampling is where teams get clever and then blind. Head-based 1% sampling will drop the exact rare checkout you needed. Tail sampling on errors and on slow traces is more work and worth it on the money path. Everywhere else, be honest about what you are throwing away.

Replay as a habit, not a postmortem slide

After a Sev-2 we ask the incoming engineer to reconstruct the request from ids alone, on a recorded call, without the person who shipped the bug. If they cannot, the work is instrumentation, not a longer incident doc. Documents do not replay Tuesday. Fields do.

We still see teams shipping `console.log` in a Node service and promising Loki “next quarter”. Next quarter the same engineer is on a new feature. Put the logger in the template repo. Make the PR check fail if a new service has no request id middleware. Culture is just defaults with teeth.

A Tuesday we could replay

Checkout id in the support ticket. Same id in the browser beacon, the API log, the queue message, the worker. Trace shows 38 seconds in a downstream KYC vendor, 200 milliseconds in us. The vendor had a regional blip. We had no timeout shorter than the user’s patience. That is an engineering change, not a mystery. Without the id, the same afternoon is a war room and a guess about “the database”.

We added a 4-second timeout and a retry with a new vendor request id. We also added the checkout id to the customer email so the next person does not have to ask engineering. That last part is product. Observability that never leaves the engineering org is a hobby.

The write-up was a page, not a novel: what we saw, what we changed, what we still cannot see. The last line is the backlog. If every postmortem ends with “improve monitoring” and no ticket, you will have this Tuesday again.

The founder dashboard

Someone who does not write code should be able to see: requests, errors, p95, queue depth, and payment success. Four or five numbers. If they have to ask you whether yesterday was fine, you built a system only engineers can love.

When Tuesday breaks, you will not reach for a strategy document. You will reach for a request id. Put that id in the product now, while you are still allowed to be calm.