✅ Confirming the Fix Worked: AI Post-Remediation Verification

I once declared an incident resolved, posted the all-clear, went back to bed — and got re-paged forty minutes later for the same thing. The “fix” had cleared the symptom I was watching (the error-rate graph dropped) but not the cause (a poisoned cache entry that re-propagated). My MTTR for that incident wasn’t the forty minutes it looked like; it was the forty minutes plus the re-detection, plus the credibility hit, plus the second investigation from scratch. Premature resolution is one of the quietest MTTR killers, and it comes from confirming the fix worked by vibes instead of by evidence.

Post-remediation verification is its own phase, and it’s underrated. AI helps by assembling the full verification picture — not just the one graph you happened to be staring at — so you close on proof.

”It looks better” is not “it’s fixed”

When you apply a mitigation under pressure, you watch the one metric that triggered the alert. It recovers, relief floods in, and you call it. The problem: incidents have more surface area than the alert that caught them. The fix might recover error rate while latency is still degraded, or recover the primary region while a replica lags, or recover the metric while the underlying resource is still leaking and will breach again in an hour. Confirming resolution means checking all of those, and a tired human checks one. This is the same correlate-many-signals problem that shows up across the MTTR funnel, now at the closing end.

Build a verification checklist from the incident itself

The right verification set is specific to this incident: the triggering SLO, the cause you confirmed during diagnosis, the blast radius you scoped, and any side effects of the fix you applied. AI is good at assembling that checklist because it can read the incident’s own record and turn it into concrete checks.

You are verifying that an incident is actually resolved. Given the original alert, the confirmed root cause, the blast radius, and the mitigation that was applied, produce a verification checklist. For each item: what to confirm, the exact query/command to confirm it, and the threshold that counts as “recovered.” Include checks for (a) the original symptom, (b) the confirmed cause, (c) the full blast radius, and (d) any side effect the fix could have introduced. Mark which checks must pass sustained over time vs. instantaneously. Do not declare resolution — produce only the checklist for a human to run.

The checklist it produces is the thing I should have had at 3 a.m.:

1. Original symptom — rate(http_5xx[5m]) < 0.01 sustained 10 min, not just instantaneous.
2. Confirmed cause (cache) — Cache hit ratio back above 0.9 and no poisoned key pattern in logs: grep 'cache_key=user:* status=stale' app.log | tail.
3. Blast radius — Recovery confirmed in all affected regions, not just us-east-1.
4. Fix side effect — The cache flush you ran spiked DB load; confirm pg connection count is back under cap and not climbing.

Run the checks, and respect “sustained”

The single most important field there is sustained vs. instantaneous. The poisoned-cache incident that re-paged me looked recovered the instant I checked because I caught it between re-propagations. A sustained check would have caught the bounce-back.

# Sustained recovery: error rate under threshold for the full window, not a single sample
curl -s "http://prom:9090/api/v1/query?query=\
max_over_time(rate(http_requests_total{status=~'5..'}[5m])[10m:])" \
  | jq -r '.data.result[0].value[1]'

# Side-effect check: is the DB connection count stable or still climbing post-flush?
kubectl exec -n payments deploy/payments -- \
  psql -tc "select count(*), now() from pg_stat_activity;" 

# Blast-radius check: confirm recovery across every region that was affected
max by (region) (rate(payment_errors_total[5m]))

If max_over_time over the last ten minutes is still showing a spike, the system bounced — it’s not fixed, and I do not post the all-clear. Evidence over hope.

AI verifies the inputs; the human declares resolved

The hard line, same as every other phase: AI assembles and runs the read-only checks and reports what it found. It does not declare the incident resolved. That’s a human judgment, because “resolved” depends on context the checks can’t fully capture — is the team comfortable, is there a follow-up risk, is the underlying fix permanent or a band-aid we’re babysitting until morning?

I treat the AI’s verification report as a pre-flight checklist for closing, and I read it adversarially:

• Every green check needs its evidence inline. “Error rate recovered” is meaningless without the number and the window. If the report can’t show me the query result, the check didn’t happen.
• A red or ambiguous check blocks resolution. No exceptions. If the side-effect check is “unclear,” the incident stays open until it’s clear.
• Re-run before you sign off, not before you start writing the all-clear. Verification has a shelf life. Run it, then immediately declare, so nothing drifts in between.

This is also where the free incident assistant is genuinely useful: paste the incident’s cause, blast radius, and the mitigation you applied, and have it generate the verification checklist before you close. The prompt library has a version with the sustained-check and side-effect framing built in.

Why this slice pays off twice

Cutting premature resolution helps MTTR in two ways. The obvious one: you stop counting “fixed” incidents that quietly reopen, so your real MTTR stops hiding behind re-detections. The subtle one: rigorous verification catches the side effects of your own fix — the cache flush that spiked the DB, the restart that lost in-flight work — before they become the next incident. A leak you catch in the verification step is an incident you never have.

I no longer trust the one graph that triggered the page to also tell me it’s over. The page caught a symptom; resolution is about the cause, the radius, and what my fix disturbed. AI assembles that full picture fast and runs the read-only checks. I read the evidence, and I’m the one who says it’s done.