Measuring Postmortem Quality: A Practical Scorecard

Teams that take postmortems seriously eventually hit a question they can’t answer: are ours any good? You can count how many you write, but volume says nothing about quality. This guide gives you a practical scorecard for rating postmortem quality, the outcome metrics that actually correlate with better reliability, and the vanity numbers that quietly fool teams into thinking they’re improving when they aren’t.

Why measure quality at all

Because postmortems degrade silently. Without a quality signal, a team drifts toward box-ticking: the template gets filled in, the meeting happens, the doc gets filed, and nobody notices the action items stopped shipping and the root causes stopped going past “human error.” A lightweight quality measure catches that drift early, and it gives you something concrete to coach on instead of a vague sense that the postmortems “feel thin.”

A postmortem quality scorecard

Rate each finished postmortem on these dimensions, 0–2 (0 = missing, 1 = present but weak, 2 = strong). It takes about five minutes per doc.

## Postmortem quality scorecard (max 20)

| Dimension                                             | 0-2 |
|-------------------------------------------------------|-----|
| Summary is clear to someone who wasn't there          |     |
| Impact is quantified (not "some users")               |     |
| Timeline is built from evidence, detection gap visible|     |
| Root cause is plural / systemic, not "human error"    |     |
| Analysis is blameless (neutral language)              |     |
| "What went well" and near-misses captured             |     |
| Action items are specific and testable                |     |
| Action items are owned + dated + tracked              |     |
| Action items span prevent / detect / mitigate         |     |
| Doc was finalized within a few days of the incident   |     |
|-------------------------------------------------------|-----|
| Total                                                 | /20 |

A score of 16+ is a strong postmortem. 10–15 is serviceable with clear gaps. Under 10 means the process is producing paperwork, not learning. The point isn’t the number — it’s the pattern. Score a quarter’s worth and the weak columns tell you exactly where to coach.

Outcome metrics that actually matter

The scorecard measures the document. These metrics measure whether the process is working:

Action-item completion rate. Of the action items opened last quarter, what fraction shipped? This is the single most honest signal. Under 50% means your postmortems are producing documents, not change.
Time-to-finalize. Median days from incident close to a finalized doc. Long lags mean decayed memory and thinner analysis.
Repeat-incident rate. How often does the same class of incident recur? A recurrence is a direct verdict on whether the earlier postmortem’s fixes worked.
Recurrence after “fixed.” Sharper: how often does an incident recur after its postmortem action items were marked done? A high number means your fixes aren’t addressing the real cause — often a root-cause-analysis quality problem.
Coverage. What fraction of qualifying incidents actually get a postmortem? If severe incidents are slipping through unreviewed, quality on the ones you write hardly matters.

The vanity metrics that fool teams

Beware numbers that look like progress but measure nothing real:

Postmortem count. Writing more postmortems is not improving. It might just mean more incidents, or more box-ticking.
Action items generated. More action items per incident is often worse — it usually means unfocused analysis that never shipped anything. Twenty items that rot beat three that ship only in the wrong direction.
Doc length. Longer is not more thorough. The best postmortems are shorter than people expect.
Meeting attendance. A packed retro can still be a blame session. Presence isn’t learning.

The tell for all of these is that they can go up while reliability goes nowhere. A metric you can improve without improving anything real is a metric that will eventually be gamed — usually without anyone intending to.

How to run the measurement without creating theater

The measurement itself can become box-ticking if you’re not careful. A few guardrails:

Sample, don’t audit everything. Score a handful of postmortems a month, not all of them. You want a signal, not a compliance regime.
Coach, don’t rank people. Use scores to improve the process and mentor authors, never to grade individuals. The moment scoring becomes a performance metric, authors write to the rubric instead of to the truth — and you’ve recreated blame with extra steps.
Review the trend, not the point. One low-scoring postmortem is noise. A quarter of declining completion rates is a signal worth a conversation.
Close your own loop. If measurement reveals that action items don’t ship, the fix is a real intervention (a standing review of open items), not a sternly-worded reminder.

Wrapping up

You can’t improve what you don’t measure, but you can easily measure the wrong thing. Rate the documents with a simple scorecard, watch the outcome metrics that correlate with real reliability — completion rate, time-to-finalize, and recurrence after “fixed” — and stay suspicious of vanity numbers like count and length that rise without anything improving. Keep the measurement lightweight and coaching-oriented, and it becomes a quiet early-warning system for a process drifting toward theater.