🚨 Incident Response Failure Mode: Gaming MTTR and the Metrics That Hide Real Pain

Overview

Leadership wants incident metrics to improve, so MTTR — mean time to recovery — becomes a target the team is measured against. Within a quarter, MTTR looks fantastic. It is also meaningless. Incidents get closed the moment the graph looks better, before recovery is confirmed. Borderline events get logged as Sev4 or not logged at all to keep them out of the average. A long, painful outage gets split into three “separate” shorter incidents. The number went down; the reliability the number was supposed to represent did not move, and in some cases got worse because the team is now managing the metric instead of the systems.

This is incident-metric gaming, a direct consequence of Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. It is a subtle and corrosive failure mode because it looks like success. Dashboards improve, leadership is pleased, and the underlying operational reality quietly diverges from the reported one — until a major incident, or a customer, reveals the gap between the metrics and the truth.

This guide covers how incident metrics get gamed, how to detect the distortion, and how to measure incident response in ways that resist manipulation.

Symptoms

• Incidents are marked resolved before recovery is actually confirmed, so the timer stops while users still hurt.
• Severity is systematically under-assigned to keep serious events out of the metrics that matter.
• Long incidents are split into several shorter ones, or short ones merged, to shape the averages.
• Some incidents are quietly not logged at all — handled “off the books” to protect the numbers.
• Reported MTTR improves while customer complaints and error budgets do not — the metric and reality diverge.
• The team talks about “the number” more than about the systems during and after incidents.
• Postmortems feel oriented toward defending the metrics rather than understanding what failed.

Common Root Causes

• A single metric turned into a target. Making MTTR (or count, or any one number) a goal the team is judged on creates direct pressure to manipulate it. This is Goodhart’s Law operating exactly as predicted.
• Metrics tied to individual or team performance evaluation. When incident numbers affect reviews or bonuses, the incentive to game them becomes personal and strong.
• Ambiguous definitions. Loose rules for when an incident starts, ends, what severity it is, and what counts as “one” incident leave wide room for favorable interpretation.
• No verification requirement for resolution. If closing an incident does not require confirmed recovery, closing early is the easy path to a better MTTR.
• Punitive incident culture. If logging an incident invites blame, people log fewer of them, and the data hollows out.
• Leadership measuring outputs instead of outcomes. Watching the vanity metric rather than customer-facing reliability rewards the appearance of improvement over the substance.
• No counter-metrics. With nothing measuring the quality and honesty of the data itself, gaming has no check.

Diagnostic Workflow

1. Compare reported metrics against independent reality. Line up MTTR and incident counts with customer complaints, error-budget burn, and support tickets. Divergence — metrics improving while pain does not — is the signature of gaming.

2. Inspect resolution-to-recovery gaps. Check whether incidents marked resolved were actually recovered at that moment, or whether users were still affected. Premature closure inflates MTTR performance.

3. Look at the severity distribution over time. A drift toward lower severities without a corresponding change in real impact suggests severity is being under-assigned to manage the numbers.

4. Hunt for split and merged incidents. Look for clusters of “separate” short incidents that were really one prolonged event, or suspiciously convenient groupings. Segmentation is a common lever for shaping averages.

5. Estimate the unlogged. Talk to on-call about events that were handled but never logged. A large shadow volume means your data describes a fiction.

6. Check what the metrics are attached to. Determine whether incident numbers feed performance reviews or bonuses. Metrics tied to evaluation are metrics under pressure to be gamed.

Example Root Cause Analysis

Incident: A quarterly review celebrated a 40% drop in MTTR. The same quarter, customer-reported outages and error-budget burn both increased. Investigation showed the MTTR improvement was almost entirely an artifact of how incidents were now being recorded.

Surface finding: “Incident response improved — MTTR dropped 40%.”

Deeper analysis:

• Premature closure: After MTTR became a tracked target, incidents were routinely marked resolved when the primary graph recovered, not when the system was verified healthy. Several “resolved” incidents reopened hours later as “new” ones.
• Severity drift: Events that would previously have been Sev2 were increasingly logged as Sev3, keeping them out of the headline MTTR figure.
• Segmentation: One prolonged degradation was recorded as four short incidents, replacing a single long duration with four favorable ones.
• Shadow incidents: On-call engineers reported handling a number of smaller events “quietly” to avoid dragging the numbers and inviting scrutiny.
• Systemic factor: MTTR had been made a target tied to team performance, with loose definitions and no verification requirement and no counter-metric checking data honesty — a textbook setup for Goodhart’s Law.

Real root cause: Not a real improvement in response, but a metric turned into a target, which incentivized premature closure, severity under-assignment, incident segmentation, and unlogged events — improving the number while reliability declined.

Corrective actions: (1) MTTR was demoted from a target to one context signal among several, and detached from individual performance evaluation. (2) Incident definitions were tightened, and resolution now requires confirmed, verified recovery. (3) Outcome measures — customer-facing reliability and error-budget burn — were elevated as the primary health signals, with data-quality counter-metrics (reopened-incident rate, logged-vs-detected ratio) to catch gaming.

Prevention Best Practices

• Never make a single incident metric the target. MTTR, count, and severity mix are useful context, not goals. The moment one becomes the objective the team is judged on, it stops describing reality. Measure a balanced set and watch trends, not a headline number.
• Prioritize outcomes over outputs. Customer-facing reliability, SLO/error-budget health, and user pain are what actually matter. Center these; treat internal process metrics as supporting detail.
• Require verified recovery to close. An incident is resolved when the system is confirmed healthy, not when a graph twitches upward. This alone removes the biggest MTTR-gaming lever.
• Tighten definitions. Clear, shared rules for incident start, end, severity, and what constitutes one incident shrink the room for favorable interpretation.
• Decouple metrics from individual evaluation. Incident data should drive learning, not performance scores. Tying it to reviews or bonuses guarantees it will be gamed.
• Add counter-metrics for data honesty. Track reopened-incident rate, resolution-to-recovery gaps, and the ratio of detected-to-logged events so gaming shows up instead of hiding.
• Build a blameless culture so incidents get logged. People log honestly when logging is safe. Punitive responses hollow out your data far more than they improve behavior.

Quick Reference

Signal	What it indicates	First action
MTTR improves, customer pain doesn’t	Metric/reality divergence	Compare metrics to error budget and complaints
Incidents closed before recovery	Premature closure	Require verified recovery to close
Severity drifting downward	Under-assignment	Tighten and audit severity definitions
One outage logged as several	Segmentation	Clarify what counts as one incident
Events handled but never logged	Shadow incidents	Build blameless logging culture
Metrics feed performance reviews	Direct gaming pressure	Decouple metrics from evaluation

Conclusion

Incident-metric gaming is dangerous precisely because it wears the mask of success. The dashboard improves, leadership relaxes, and the reported story of your reliability drifts away from the real one — until a major incident or a frustrated customer exposes the gap. Goodhart’s Law is not a cynical observation; it is a reliable prediction. Make MTTR a target and you will get better MTTR numbers and no better reliability, because the team is now optimizing the measurement, not the system.

Measure what you actually care about: verified recovery, customer-facing reliability, and error-budget health, watched as a balanced set of trends rather than a single headline number. Decouple the data from performance scoring, tighten your definitions, add counter-metrics that catch manipulation, and keep incident logging blameless so the data stays honest. A metric is only worth having if it still tells the truth once people know you are watching it.