🚨 Postmortem Anti-Pattern: Remediations That Are Never Verified

Overview

The most satisfying moment in the incident lifecycle is checking off the last action item. The postmortem is written, the remediations are agreed, the tickets get worked, and one by one they close. The dashboard turns green, the tracking spreadsheet shows 100% complete, and everyone moves on with the comfortable belief that this failure has been handled. But “the ticket is closed” and “the failure can no longer happen” are two entirely different claims, and this anti-pattern is the habit of treating the first as if it proves the second. A remediation is a hypothesis — “this change will prevent the failure” — and a hypothesis that is never tested is just an assumption wearing a completed checkbox.

This gap between remediation-done and remediation-proven is where false confidence lives, and false confidence is more dangerous than known uncertainty because it stops you from looking. A retry that was added but never exercised under real failure, a config change that fixed the symptom in the moment but was never validated against the original trigger, an alert that was created but never test-fired — each of these looks like protection and provides none. The organization believes it has closed a class of incident when it has merely closed a set of tickets, and the belief holds right up until the identical failure recurs and everyone asks how, when “we fixed that.” The answer is that fixing was never confirmed. This guide is about closing that gap: treating verification, not ticket closure, as the definition of done.

Symptoms

• Action items closed as “done” with no evidence attached — a ticket marked complete but nothing showing the fix actually behaves as intended.
• “Should” language in the postmortem — “this change should prevent recurrence” — with no test, game day, or observation cited to turn “should” into “does.”
• Alerts and runbooks created but never fired or rehearsed, so nobody knows whether the alert triggers on the real condition or the runbook actually works under pressure.
• Fixes validated only against the symptom, not the trigger — the change made the current error go away, but was never tested against the original failure scenario.
• No owner for verification, distinct from the owner who implemented the change — the person who wrote the fix marks their own homework, if anyone checks at all.
• Recurrence of an incident whose remediations were all marked complete, revealing the fixes never held.
• A completion metric reported upward — “all action items closed” — with no accompanying claim that the failure mode was demonstrably eliminated.
• Reluctance to deliberately trigger the failure to see if the fix holds, because doing so feels risky or unnecessary once the ticket is closed.

Common Root Causes

• “Closed” conflated with “working.” The process defines done as ticket-closed, so implementation is where the effort stops and verification never gets scheduled.
• No verification step in the workflow. The remediation lifecycle ends at “implemented” with no explicit “proven” stage, so proving is nobody’s job.
• Verification feels expensive or scary. Deliberately reproducing a failure to confirm the fix holds takes effort and nerve, so it’s skipped in favor of assuming.
• The implementer verifies their own work. Without an independent check, confirmation bias lets a plausible-looking change pass as a proven one.
• Symptom-level thinking. Responders validate that the immediate error cleared, mistaking symptom relief for elimination of the underlying trigger.
• No safe place to test failure. Without a staging environment, fault-injection tooling, or a game-day practice, there’s often no realistic way to exercise the fix even when someone wants to.
• Pressure to show closure. Reporting “100% of action items complete” is rewarded; reporting “implemented but unverified” is not, so the incentive favors closing over confirming.

Diagnostic Workflow

For every remediation, separate the claim from the evidence. The action item asserts that a change prevents a failure; verification is the act of producing evidence for that assertion. Closing a ticket produces no evidence.

For each action item, ask:
  What EXACTLY does this claim to prevent?  (name the failure mode)
  What evidence shows it prevents it?       (test | game day | observation)

If the only "evidence" is "the ticket is closed" -> UNVERIFIED.
An unverified remediation is an assumption, not a fix.

Then design a verification that targets the original trigger, not the symptom you happened to see during the incident. Reproducing the actual failure condition is the only test that proves anything.

Verification must exercise the ORIGINAL trigger:
  reproduce the failure condition (fault injection / game day / staging replay)
  confirm the remediation activates and prevents the impact
  for alerts   -> deliberately create the condition; confirm it fires
  for runbooks -> have someone unfamiliar execute it end to end
  for retries/failover -> inject the fault; watch it actually recover

Symptom cleared != trigger neutralized. Test the trigger.

Finally, assign verification to someone other than the implementer, and require evidence — a game-day writeup, a test result, an alert-fired screenshot — to be attached before the action item is allowed to close. Make “done” mean “proven,” and make proof independent. Where a remediation genuinely cannot be tested safely yet, that fact should be recorded as a known risk rather than hidden behind a green checkbox.

Example Root Cause Analysis

Incident: The API tier fell over for 35 minutes when a downstream identity provider (IdP) slowed to a crawl; requests piled up waiting on IdP calls until the API’s thread pool was exhausted. The team added a timeout and a circuit breaker on the IdP client, and closed both action items.

Unverified version: “Remediation: added a 2-second timeout and a circuit breaker to the IdP client. Both deployed to production. Action items closed. This should prevent thread-pool exhaustion if the IdP slows down again.” Status: 100% complete, marked resolved. Nobody has actually made the IdP slow to confirm the new behavior.

Why this is wrong: Every claim in that remediation is a hypothesis. Nobody has demonstrated that the timeout fires at 2 seconds under real latency, that the circuit breaker opens before the thread pool saturates, or that the API sheds load gracefully instead of failing a different way. The change was validated only against the symptom’s absence — the current traffic is healthy, so it looks fine — not against the original trigger, a slow IdP. A misconfigured timeout that never actually engages, or a circuit breaker whose threshold is set higher than the pool can absorb, would look identical in this state: green, closed, “done.” The team believes it has eliminated the failure mode when it has only deployed an untested guess about it.

What verification reveals: Running a game day that deliberately injects IdP latency shows the timeout does fire — but the circuit breaker’s failure threshold is set so high that the thread pool still exhausts before the breaker opens. The fix, as shipped, would not have prevented the original incident. This is exactly the finding a closed ticket would have hidden until the next real IdP slowdown.

Corrected version: Keep the remediations open until proven. Run a controlled fault-injection game day that reproduces the IdP slowdown in staging; tune the circuit-breaker threshold below the pool-exhaustion point based on what the game day shows; re-run to confirm the API now sheds load gracefully; attach the game-day results as evidence. Real definition of done: the failure was reproduced and the remediation was observed to prevent it, verified by someone other than the implementer, with evidence attached. Only then do the action items close — and now “we fixed that” is a fact rather than a hope.

Prevention Best Practices

• Redefine “done” as “proven,” not “implemented.” An action item closes only when there is evidence the failure mode can no longer occur, not when the code merged.
• Add an explicit verification stage. Give every remediation an implemented-then-verified lifecycle so proving the fix is a required, scheduled step rather than an optional afterthought.
• Verify against the original trigger. Reproduce the actual failure condition — via fault injection, a game day, or a staging replay — rather than relying on the symptom having cleared.
• Separate verifier from implementer. Have someone independent confirm the fix, so nobody marks their own homework and confirmation bias has less room to hide a non-fix.
• Require attached evidence to close. A game-day writeup, a test result, or an alert-fired confirmation must accompany closure; “the ticket is closed” is not evidence.
• Rehearse alerts and runbooks, don’t just write them. Test-fire every new alert against its real condition and have an unfamiliar engineer execute every new runbook end to end.
• Record untestable remediations as known risk. When a fix genuinely can’t be verified safely yet, log that honestly as residual risk instead of concealing the uncertainty behind a green checkbox.

Quick Reference

# Anti-pattern: remediations never verified -> "closed" mistaken for "works."
# A remediation is a hypothesis; untested, it's an assumption in a checkbox.

For each action item:
  What failure mode does it claim to prevent?
  What EVIDENCE proves it prevents it? (test | game day | observation)
  Only evidence is "ticket closed" -> UNVERIFIED.

Verify against the TRIGGER, not the symptom:
  reproduce the original failure condition (fault injection / staging replay)
  confirm the remediation activates and prevents the impact
  alerts   -> create the condition, confirm it fires
  runbooks -> unfamiliar engineer runs it end to end
  retry/failover -> inject the fault, watch it recover

Rules:
- "done" = PROVEN, not implemented
- verifier != implementer
- evidence attached before close
- untestable-for-now -> record as known risk, don't hide it

Conclusion

Remediations that are never verified are seductive because they generate the feeling of safety without the substance of it: the tickets close, the metric hits 100%, and everyone believes a class of incident has been eliminated. But a remediation is only a hypothesis about how the system will behave under a failure it is not currently experiencing, and an untested hypothesis is indistinguishable — in a healthy, green, closed-out state — from one that would collapse the moment the real trigger returns. That false confidence is worse than acknowledged uncertainty, because it tells you to stop looking at precisely the thing most likely to page you again. The fix is to make verification the definition of done: reproduce the original trigger, confirm the remediation actually holds, have someone independent check it, and attach the evidence before anything closes. Closing a ticket ends a task; proving a fix ends an incident. Until you have watched the failure try and fail to recur, you have not remediated it — you have only hoped.