📡 Cutting Time-to-Detect With AI Anomaly Summarization

At 02:14 the dashboards went from green to a wall of red. Eleven alerts fired inside ninety seconds, three of them paging, and the on-call’s first two minutes were spent not investigating but parsing — which of these is the cause and which are echoes? By the time anyone could say “checkout latency moved first, at 02:11, the rest followed,” the incident was four minutes old and we hadn’t started diagnosing anything. That gap between the first real deviation and a human understanding it is time-to-detect, and it’s the slice of MTTR nobody puts on the retro slide.

Most teams instrument detection well enough that something fires. The expensive part is comprehension: turning a noisy monitoring surface into “here is what changed, when, and what’s leading versus trailing.” That’s a summarization problem, and it’s one AI is genuinely good at — if you keep it on summarizing and off diagnosing.

Detection isn’t the alert, it’s the understanding

We conflate “an alert fired” with “we detected the incident,” but those are different moments. The alert is a threshold crossing. Detection is a human knowing what’s happening well enough to act. When a cascade lights up the board, the threshold crossings arrive in a heap with no ordering, and the responder has to reconstruct the sequence by hand. That reconstruction is recoverable time, and it’s the same structural problem that shows up all across the MTTR funnel: speed comes from organizing the signal, not from staring harder.

A model reading your firing alerts and key series doesn’t have to scroll. Ask it for a timeline ordered by first deviation, with leading-edge signal separated from downstream noise, and you’ve replaced two minutes of frantic parsing with a structured picture.

Ask for a ranked timeline, not a diagnosis

The framing matters. If you ask “what’s wrong?”, you’ve skipped detection and jumped to diagnosis on incomplete data. Ask instead for the shape and order of what changed.

You are summarizing a noisy monitoring surface. Given these firing alerts with first-seen timestamps and these metric series, produce a timeline ordered by when each anomaly first deviated from baseline. For each: the shape (step / ramp / spike / oscillation), whether it’s correlated across services or isolated, and whether it’s a leading-edge signal or a downstream symptom. Flag any metric that moved before any alert fired. Do not name a root cause. Mark where missing baseline makes signal indistinguishable from normal variance.

The output reorganizes the chaos into a sequence:

| Time | Anomaly | Shape | Lead/secondary | | 02:11 | checkout p99 latency 3x | ramp | leading | | 02:12 | cart-service error rate up | step | secondary (waits on checkout) | | 02:13 | API gateway 5xx | spike | secondary | | 02:13 | DB connection saturation | ramp | leading? (independent — confirm) |

Earliest real deviation: checkout p99 at ~02:11, three minutes before the first page.

Now the responder isn’t parsing eleven alerts — they’re looking at two leading candidates and a clear start time, three minutes earlier than the page suggested.

Confirm the start time with a read-only query

The summary is a hypothesis about when things began. Confirm it before you build on it:

# Pull the true onset for the leading-edge metric, read-only
curl -s "http://prom:9090/api/v1/query_range?query=\
histogram_quantile(0.99,sum(rate(http_request_duration_seconds_bucket\
{service=\"checkout\"}[1m]))by(le))&start=$(date -d '20 min ago' +%s)\
&end=$(date +%s)&step=30" \
  | jq -r '.data.result[].values[] | "\(.[0]) \(.[1])"'

# Sanity-check the suspected independent signal against baseline
kubectl exec -n data deploy/pg-bouncer -- \
  psql -tc "select count(*) from pg_stat_activity;" 

In the 02:14 incident, confirming the latency onset at 02:11 mattered: it pointed diagnosis at a deploy that landed at 02:10, which a 02:14-anchored investigation would have ranked as “after the alert, so irrelevant.”

Keep the AI on detection, not conclusions

The non-negotiable: anomaly summarization narrows where to look and when it started, never why. The moment the model offers a cause, you’ve reintroduced the anchoring problem the timeline was meant to prevent — and you’ve done it on the thinnest possible evidence, the opening seconds of an incident.

A few rules that keep this honest:

• Treat “earliest deviation” as a claim to confirm. Without good baselines, the model will happily label seasonal variance as an anomaly. The read-only query is what turns the claim into a fact.
• Don’t let a “secondary” label suppress a real second failure. A genuinely independent fault can hide behind a cascade. Before dismissing a demoted alert, check it shares a causal chain with the leading edge.
• Re-summarize as new signal arrives. The first timeline is built on partial data; regenerate it when the picture changes rather than clinging to the opening read.

You can practice this on the free incident assistant — paste a real burst of alerts and metric notes and ask for the ranked timeline, then notice how having the leading edge separated from the noise changes where you look first. The prompt library has a hardened anomaly-summarizer prompt with the detection-gap framing built in.

Time-to-detect is the slice of MTTR you can’t see until you measure it, and the fastest way to shrink it isn’t more alerts — it’s faster comprehension of the alerts you already have. AI is well-suited to turning a wall of red into a ranked timeline in seconds, as long as it stops at what changed and when and leaves why to the humans with their hands on the read-only queries.