Postmortem Anti-Pattern: Treating Every Incident as Brand New
Every incident feels novel until you notice it isn't. How failing to link an incident to its prior siblings lets the same failure recur forever unseen.
- #postmortems
- #incident-response
- #sre
- #troubleshooting
Stuck on this Post Mortems with AI error? Get the free incident triage checklist
A one-page PDF — the exact steps to isolate, fix, and verify a production error like this one. No spam, unsubscribe anytime.
Overview
Every incident arrives feeling urgent and singular. The pager fires, a team scrambles, the symptom is real, and the pressure to restore service crowds out everything else — including the question of whether this exact thing has happened before. So the responders fix the immediate problem, write a postmortem scoped tightly to this one event, close the action items, and move on. The next time the same failure mode surfaces, a different on-call engineer meets it as if for the first time, rediscovers the same diagnosis, and files another standalone report. The organization is running a treadmill it cannot see.
This anti-pattern is corrosive because the cost is invisible in any single incident. Each individual postmortem can be competent, accurate, and blameless, and the process can still be quietly failing. Recurrence is a property of the sequence, not the event, and a process that only ever looks at one event at a time is structurally blind to it. The same root cause gets “fixed” three, five, ten times because nobody connects the dots, and the real fix — the one that would end the pattern — never gets prioritized because no one is looking at the aggregate. The goal of this guide is to make the pattern visible: to treat linkage to prior incidents as a first-class step, not an afterthought.
Symptoms
- A strong sense of déjà vu during response — a veteran responder mutters “haven’t we seen this before?” but nobody has the reference to hand.
- Repeated postmortems with near-identical timelines, differing mainly in the date and the name of the on-call engineer.
- Action items that read like earlier ones, sometimes verbatim, sometimes closed as “already done” from a prior incident that clearly didn’t hold.
- No “related incidents” field in the postmortem template, so linkage has no place to live even when someone notices it.
- Rediscovery of the same diagnosis from scratch each time, burning response minutes on a problem the team has already solved before.
- Metrics that look fine per-incident — good MTTR, tidy reports — while the same service keeps paging.
- Institutional knowledge locked in individuals, where only the one engineer who saw it last time recognizes the pattern, and only if they happen to be online.
- A backlog full of “quick mitigations” for one service, none of which is the structural fix, because each was scoped to a single occurrence.
Common Root Causes
- No incident-linking step in the process. The template and the review meeting never ask “has this happened before,” so the question is never answered.
- Search is hard or absent. Past postmortems live in scattered docs, tickets, or chat threads with no consistent tagging, so finding the prior sibling costs more than rediscovering the fix.
- Response pressure crowds out retrospection. Restoring service is urgent; checking history feels optional, so under load it gets skipped every time.
- On-call rotation spreads incidents across people. The same failure hits five different engineers over six months, and no single person accumulates enough exposures to see the trend.
- Mitigations masquerade as fixes. A restart or a config bump makes the symptom go away, the incident closes green, and the underlying condition survives to recur.
- No aggregate review. Nobody periodically looks across incidents for clusters, so patterns that are obvious in bulk stay invisible one at a time.
- Fear of admitting recurrence. Linking to a prior incident can feel like admitting the last fix failed, so the incentive quietly favors filing a fresh, unconnected report.
Diagnostic Workflow
Before scoping a postmortem as novel, run a deliberate prior-art search. The question is not “what fixed this” but “have we seen this signature before, and what did we conclude then?”
For the incident's signature, search prior postmortems/tickets by:
affected service / component
error message or exception signature
alert name that fired
symptom shape (latency spike | error rate | saturation | data drift)
time-of-day / deploy correlation
If ANY prior match exists:
-> this is NOT a new incident. Link it. Treat as a recurrence.
For each candidate match, decide whether it is genuinely the same failure mode or a superficial look-alike. Same symptom does not always mean same cause.
Same failure mode if:
same root cause OR same contributing conditions
AND the prior action items were meant to prevent exactly this
If same failure mode AND prior action items were "completed":
-> the prior fix did not hold. Ask WHY it didn't, not just what to do now.
Then widen the lens beyond the single event. On a regular cadence — monthly is a reasonable start — cluster recent incidents and count repeat offenders. A failure that has recurred three times is not three small incidents; it is one unresolved problem wearing three costumes. Rank by frequency times impact, and let that ranking, not the recency of the last page, decide what gets the structural fix.
Example Root Cause Analysis
Incident: The checkout service returned elevated 500s for 18 minutes after a background job exhausted the database connection pool. On-call scaled the pool, errors cleared, incident closed.
Novel-incident version: “Root cause: the reporting job opened too many connections and exhausted the pool. Fix: increased pool size from 50 to 80. Action item: monitor pool utilization.” The report is scoped entirely to this occurrence and references no history.
Why this is wrong: A search of prior postmortems would have surfaced two earlier incidents — one three months ago, one seven — with the same signature: a background job, pool exhaustion, checkout 500s. Both had been “fixed” by raising the pool size. This is the third time the same structural problem has been mitigated by adding headroom, and the headroom keeps getting eaten. Treating it as new means applying the same non-fix a third time and setting up a fourth incident.
The linkage that was missed: All three share a root cause the per-incident view never reached: background jobs and user-facing traffic draw from the same unbounded connection pool, so any heavy job can starve checkout. Raising the pool ceiling delays the next exhaustion; it does not prevent it.
Corrected version: “Root cause (recurring — see INC-1041, INC-1088): user-facing and batch workloads share a single unbounded connection pool with no isolation, allowing background jobs to starve checkout. This is the third recurrence; prior mitigations (pool size increases) addressed symptoms, not the shared-pool design.” Real action items: give batch jobs a separate, bounded pool isolated from user traffic; add a per-workload connection cap; add an aggregate review that flags any incident recurring three or more times for structural remediation. The report links its siblings and treats the sequence, not the event, as the thing to fix.
Prevention Best Practices
- Make “related incidents” a required field. No postmortem ships without an explicit prior-art search, even if the honest answer is “no prior match found.”
- Tag incidents with a stable signature. Consistent tags for service, error signature, and symptom shape make the prior-art search cheap enough to actually do under pressure.
- Run a periodic aggregate review. Monthly, cluster recent incidents and surface repeat offenders; rank structural fixes by frequency times impact, not by which paged most recently.
- Distinguish mitigations from fixes explicitly. Label each action item as “mitigation” or “structural fix,” and track whether the structural fix ever ships — a mitigation-only incident is unfinished.
- Reframe recurrence as a signal, not an embarrassment. When a fix didn’t hold, the useful question is why it didn’t, which keeps people willing to link honestly instead of filing fresh reports.
- Escalate on the Nth recurrence. Define a threshold — say three — at which a recurring failure automatically earns a dedicated structural-fix owner and priority.
- Keep postmortems searchable in one place. Scattered knowledge is unfindable knowledge; a single indexed, tagged store is what makes linkage possible at 3 a.m.
Quick Reference
# Anti-pattern: treating every incident as brand new -> the same failure recurs unseen.
Before scoping a postmortem as novel, search prior art by:
service | error signature | alert name | symptom shape | deploy/time correlation
Any prior match -> NOT new. Link it. Treat as recurrence.
Same failure mode test:
same cause/conditions AND prior action items meant to prevent this?
If prior items were "done" -> the fix didn't hold. Ask WHY.
Recurrence handling:
- mitigation vs structural fix -> label every action item; ship the structural one
- Nth recurrence (e.g. 3) -> assign a dedicated owner + priority
- per-incident metrics look fine -> run monthly AGGREGATE review for clusters
- knowledge in one person's head -> tag + index so anyone can find the sibling
Conclusion
The danger of treating every incident as brand new is that it hides in plain competence: each report can be well written and each fix can technically work, and the same failure can still recur forever. Recurrence is a property of the sequence, and a process that only inspects one event at a time cannot detect it. Break the pattern by making linkage a first-class step — a required prior-art search, stable signatures, and a regular aggregate review that ranks repeat offenders for structural fixes rather than yet another mitigation. When responders can see that this failure has three siblings, “raise the pool size again” stops looking like a fix and starts looking like what it is: the fourth incident, pre-scheduled. The point of a postmortem is not just to explain one outage but to end the class of outage it belongs to.
Fixed it? Get 500 Post Mortems with AI & DevOps AI prompts — free
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.
Did this fix your issue?
Get 500 Battle-Tested DevOps AI Prompts — Free
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.