🧵 Narrowing Scope With AI Log and Trace Correlation

The trace was right there on the screen — a waterfall fourteen spans deep, half of them tinted red — and I still couldn’t tell you in under a minute where the time was actually going. Three services looked slow, but two of them were slow only because they were blocked waiting on the third. Then there were the logs: a thousand lines in the same five-second window, most belonging to requests that had nothing to do with this one. Diagnosis wasn’t blocked on missing data. It was blocked on stitching — correlating the trace to the logs to find the failing path. That stitching is where diagnosis minutes quietly disappear.

The bottleneck and the relevant log lines are almost always already in the data. The work is the correlation, and that’s a job a model can do explicitly: name the span where time is truly spent, then pull the log lines that belong to that exact path.

The hot path hides behind the spans waiting on it

A trace waterfall is deceptive because waiting looks like working. A span shows 800ms not because it’s doing 800ms of work but because it’s blocked on a downstream call that is. Eyeball the waterfall and you’ll suspect the wrong service. The skill is distinguishing the span where time is genuinely spent from the spans merely waiting on it — and then finding the logs from that span, not the logs that merely share a timestamp. This is the same narrowing instinct the rest of the MTTR funnel rewards: converge on the failing path, don’t widen the search.

A model handed a trace and a log excerpt can do this correlation in one pass, linking by trace and request IDs rather than timestamp coincidence, which is the part that’s slow and error-prone by hand.

Ask for the bottleneck and the next filter

The framing keeps it to narrowing, never root cause.

You are narrowing an incident by correlating this trace with these logs, not declaring root cause. From the trace, name the span where time is actually spent or the error originates, and distinguish it from spans that are slow only because they wait on it. Match the pasted log lines to that span using trace/request IDs, separating path-relevant lines from same-window noise. State the narrowed scope in one sentence. Give the single most decisive read-only query to pull the full matching set. Flag where sampling or missing context propagation makes this partial.

The output points you at the path:

Bottleneck span: inventory-service / reserveStock — 740ms of the 820ms request; checkout and api-gateway are slow only because they await it. Path-relevant logs: lines with trace_id=a3f9… show reserveStock retrying a locked row 4x before timing out. Other red lines in the window belong to unrelated trace_ids — noise. Narrowed scope: failing path is checkout → inventory-service.reserveStock; time is in row-lock contention; logs show retry-on-lock. Next query: pull every span for reserveStock in the last 10 min where duration > 500ms.

You’ve gone from “checkout is slow” to “reserveStock is contending on a row lock” without scrolling a thousand log lines by hand.

Pull the full set with one read-only query

The correlation gives you a lead and the single query that confirms it across the full population, not just the sample you pasted:

# Pull all slow reserveStock spans (read-only trace query)
curl -s "http://tempo:3200/api/search?tags=\
service.name=inventory-service%20name=reserveStock&minDuration=500ms&limit=50" \
  | jq -r '.traces[] | "\(.traceID) \(.durationMs)ms"'

# Confirm the lock-contention story in logs, scoped to the path
logcli query '{service="inventory-service"} |= "reserveStock" |= "lock"' \
  --since=10m --limit=100

In the checkout incident, the trace query came back with forty slow reserveStock spans, all in the last eight minutes, all on the same hot SKU — confirming that the bottleneck the single trace suggested was a real, widespread pattern and not a one-off slow request.

A bottleneck is a lead, not a verdict

The discipline: trace correlation narrows where the time is and what path it’s on, never why. And it does so from imperfect data. Sampling means the span you were handed may not be the slowest one in the population; broken context propagation means some path-relevant logs won’t carry the trace ID. So the identified bottleneck is “pull the full set and confirm,” not a proven hot path.

Rules I hold to:

• Link logs by ID, not by clock. Same-window-but-unlinked lines are noise; treating them as evidence is how you narrow toward the wrong path.
• Confirm the sample is representative. One slow trace is a hypothesis; the read-only query that returns the full slow population is what makes it real.
• Stop at the path, hand off to diagnosis. Once you know it’s reserveStock row-lock contention, that’s a narrowed scope to diagnose — not a root cause to act on.

You can practice this on the free incident assistant — paste a trace and a log excerpt and ask for the bottleneck span plus the next filter, then notice how separating the hot span from the spans waiting on it changes where you look. The prompt library has a hardened log-and-trace correlation prompt with the ID-linking and sampling caveats built in.

Diagnosis drags when you’re stitching telemetry by hand, and that stitching is recoverable MTTR. AI correlates the trace to the logs in one pass, names the span where time is actually spent, and hands you the one query that confirms it across the full population — narrowing the search to the failing path while leaving the verdict, as always, to the human running the query.