🚨 Postmortem Anti-Pattern: MTTR Metrics Theater

Overview

Reliability metrics exist to tell you the truth about your systems: how often they break, how long they stay broken, how much users suffer. That is enormously valuable — right up until the number stops being a thermometer and becomes a scoreboard. The moment mean time to recovery (MTTR) or incident count is tied to a team’s reputation, a manager’s bonus, or a slide in the quarterly review, people begin, often without any bad intent, to optimize the number rather than the reality it was meant to reflect. The metric goes green while the systems get no more reliable, and sometimes get worse.

This is metrics theater: the performance of reliability in place of the practice of it. It is a direct application of Goodhart’s Law — when a measure becomes a target, it ceases to be a good measure — and it is especially insidious in the postmortem process, because postmortems are where the raw data of an incident gets recorded. If the recording is bent to protect a metric, every downstream analysis inherits the distortion. Teams learn to classify severity down, to start the clock late and stop it early, to avoid declaring incidents at all. The dashboard becomes a work of fiction that everyone agrees to believe. This guide is about spotting the theater and getting back to metrics as signal.

Symptoms

• Suspiciously smooth MTTR that trends steadily downward regardless of the actual variety and severity of incidents — real reliability is lumpy; gamed reliability is polished.
• Severity downgrades under discussion, where the debate about whether something is a SEV-2 or SEV-3 is driven by how it will look, not by user impact.
• Incidents that never get declared — “let’s just fix it quietly” — so they never enter the count in the first place.
• Clock manipulation, where the incident is marked “resolved” the instant the symptom clears but the real fix is hours away, or the start time is quietly set to when someone got around to opening the ticket.
• Postmortems that celebrate the metric — “great job, we hit our MTTR target” — more than they examine what actually failed.
• Reluctance to log near-misses or degradations because they might count against the numbers.
• A widening gap between the dashboard and lived experience, where on-call engineers are exhausted and users are complaining while the reliability slide is all green.
• Metrics reported upward without context — a single MTTR number with no distribution, no severity breakdown, no mention of what was learned.

Common Root Causes

• Metrics tied to incentives. When bonuses, promotions, or team reputation ride on a number, the number will be optimized — this is human, predictable, and mostly not malicious.
• Single headline metrics. A lone MTTR or incident-count figure is trivially gameable; it collapses a rich, messy reality into one number that hides more than it shows.
• Averages that hide the tail. Mean time to recovery is dominated by outliers, so a few quick auto-recoveries can bury the one four-hour outage that actually mattered.
• Ambiguous definitions. If “resolved” and “incident” aren’t rigorously defined, the clock and the count become matters of interpretation — and interpretation drifts toward whatever looks best.
• Metrics owned by the measured. When the team being evaluated also classifies severity and stamps the timestamps, the fox is guarding the henhouse.
• Leadership that reads dashboards, not postmortems. If the top of the org only ever sees the aggregate number, the incentive is to make the number pretty rather than to make the writeup honest.
• No counter-metric for honesty. Nothing rewards declaring more incidents or logging near-misses, so the system quietly punishes transparency.

Diagnostic Workflow

Start by asking what each metric is being used for. A number consumed for learning and a number consumed for judgment create opposite incentives, and the same MTTR figure is healthy in the first context and toxic in the second.

For each reliability metric, ask:
  Who sees it, and what happens because of it?
  -> used to spot trends and prioritize fixes   = signal (healthy)
  -> used to evaluate/reward/punish a team      = target (Goodhart risk)

If it's a target, expect gaming. Look for it.

Then audit the definitions and the edges, because that is where theater lives. Vague boundaries are the raw material of a gamed number.

Check the definitions:
  "Incident"  - what threshold forces a declaration? who decides?
  "Resolved"  - symptom cleared, or root cause fixed? who stamps it?
  Severity    - objective user-impact criteria, or negotiable?

Red flags:
  resolved-time = symptom-clear-time, always      -> clock stopped early
  start-time = ticket-open-time, not detect-time  -> clock started late
  severity debated on optics, not user impact     -> classification gaming
  incidents "handled quietly," never declared     -> count suppression

Finally, look past the average to the distribution and the ground truth. Pull the full spread of recovery times, not the mean; overlay severity; and cross-check the dashboard against a source the metric can’t edit — user complaints, error budgets, on-call hours, support tickets. A green dashboard sitting on top of exhausted engineers and unhappy users is the definitive signature of theater.

Example Root Cause Analysis

Incident: A payments degradation caused intermittent transaction failures for roughly 90 minutes. Auto-retry masked most user-visible errors after the first 15, but the underlying broker stayed unhealthy until an engineer manually failed it over.

Metrics-theater version: The incident is logged as SEV-3 (not SEV-2, “since retries caught most of it”). Start time is stamped when the engineer opened the ticket, 20 minutes after the first alert. Resolved time is stamped at the 15-minute mark, when user-visible errors dropped, even though the broker wasn’t fixed for another 75. Recorded MTTR: “under 10 minutes.” The postmortem’s headline: “MTTR target met; retry logic performed as designed.”

Why this is wrong: Every number in that record was shaped to protect the metric. Severity was downgraded on optics, not the real 90-minute user impact; the start clock was set late to shave detection latency; the resolved clock was stopped at symptom-masking, not root-cause repair. The “under 10 minutes” MTTR is fiction — the system was genuinely unhealthy for an hour and a half — and the postmortem congratulates the number instead of examining why the broker failed and why failover was manual.

What honest measurement shows: true detection-to-resolution was ~90 minutes; retries hid impact but did not resolve it; failover required a human and took most of the incident. The distribution, not the average, is the story — this is a long-tail incident that a mean would have buried anyway.

Corrected version: Log the real severity (SEV-2, sustained payments impact), the real timeline (detect time to root-cause-fix time = ~90 min), and report MTTR as a distribution with this outlier called out. Real action items: automate broker failover so recovery isn’t gated on a human; add root-cause-fixed (not just symptom-cleared) as the official resolved timestamp; move severity classification to an incident commander independent of the owning team; report reliability as a distribution plus a learning summary, never a single averaged number.

Prevention Best Practices

• Separate learning metrics from evaluation metrics. Use reliability numbers to find and prioritize problems; never tie an individual’s or team’s reward directly to MTTR or incident count.
• Report distributions, not averages. Show the spread and the tail; a single mean is the easiest number to game and the most misleading to trust.
• Define “incident” and “resolved” rigorously. Resolved should mean root cause addressed, not symptom masked, and the definitions should be owned centrally, not by the team being measured.
• Decouple classification from the owning team. Have an independent incident commander or review function set severity and timestamps so the measured don’t grade themselves.
• Add a counter-metric for honesty. Track and reward declared near-misses and degradations, so transparency is an asset rather than a liability.
• Read postmortems, not just dashboards. Leadership should engage with the narrative and the learning; when the top of the org reads writeups, the incentive shifts from pretty numbers to honest ones.
• Cross-check against ground truth. Regularly reconcile the dashboard with user complaints, error budgets, and on-call load; a persistent gap between them is your theater alarm.

Quick Reference

# Anti-pattern: MTTR metrics theater -> optimizing the number, not the reliability.
# Goodhart: when a measure becomes a target, it stops being a good measure.

Signal vs target:
  metric used for learning/prioritizing  = healthy signal
  metric used to reward/punish a team    = target -> expect gaming

Where theater hides (audit these):
  "resolved" = symptom cleared, not root cause fixed  -> clock stopped early
  start = ticket-open, not detect time                -> clock started late
  severity argued on optics, not user impact          -> classification gaming
  incidents "handled quietly," never declared         -> count suppression
  single averaged MTTR, no distribution               -> tail hidden

Fixes:
- report distributions + tail, not the mean
- rigorous "incident"/"resolved" definitions, owned centrally
- classification by an INDEPENDENT commander, not the measured team
- add a counter-metric that REWARDS declaring near-misses
- read postmortems upward, not just dashboards
- cross-check dashboard vs user complaints / on-call hours

Conclusion

Metrics theater is dangerous because it produces exactly the reassurance leadership wants to see while hollowing out the reliability that reassurance is supposed to represent. The failure is systemic, not personal: attach a consequence to a single number and rational people will optimize the number, bending severity, clocks, and counts until the dashboard is a performance rather than a reading. The way out is to change what the metrics are for — separate learning from evaluation, report distributions instead of gameable averages, define incident and resolved rigorously, move classification to an independent hand, and reward the honesty of declaring near-misses. Above all, read the postmortems, because the narrative is far harder to fake than the aggregate. A reliability metric earns its keep only when it can still deliver bad news; the day it can no longer surprise you is the day it stopped being a signal and became a show.