🚨 Manual Rollback With No Runbook: An MTTR Failure Mode

Overview

The manual-rollback anti-pattern is the failure mode where a deploy causes an incident, everyone agrees the fix is to roll back — and then nobody knows exactly how. There is no runbook, no one-command action, and no rehearsed procedure. The rollback becomes an improvised, high-adrenaline scramble: someone tries to remember the last known-good version, someone else worries about a database migration that may not reverse, and the incident commander watches the recovery phase stretch out while a mitigation that should take 90 seconds takes 40 minutes.

This anti-pattern attacks the last and most impactful phase of MTTR: time-to-recover. It is especially damaging because it strikes when the fix is already known. The team has correctly diagnosed the problem — a bad deploy — and the remaining time is spent not on figuring out what to do but on nervously working out how to do it safely. Worse, an improvised rollback under pressure is exactly the situation where a mistake causes data loss and turns a short outage into a long one. Reducing MTTR for deploy-caused incidents means making rollback a boring, known, one-command action.

Symptoms

• “How do we roll this back?” mid-incident. The recovery phase opens with the team figuring out the rollback procedure from scratch.
• Hunting for the last known-good version. Nobody is sure which artifact or commit was healthy before the bad deploy.
• Migration panic. The team freezes because a schema migration shipped with the deploy and no one knows if rolling back the code is safe against the new schema.
• Copy-pasted commands from Slack history. The rollback is assembled from fragments of past incidents rather than a maintained runbook.
• Rollbacks that make things worse. An improvised rollback causes data loss, a second outage, or leaves the system in a half-reverted state.
• Wide variance in recovery time. Similar deploy incidents take wildly different times to mitigate depending on who is on call.

Common Root Causes

• No rollback runbook. The procedure exists only as folklore; it was never written down, versioned, or tested.
• Irreversible migrations shipped with code. Destructive schema changes (dropped columns, non-additive migrations) are deployed alongside the code, so code rollback is unsafe.
• No pinned known-good version. The release system does not record or expose the previous healthy artifact for one-step reversion.
• State that doesn’t roll back. Feature-flag state, caches, queues, and dual-writes are left behind by a code rollback and nobody planned for them.
• Rollback never rehearsed. The procedure has not been practiced in staging or a game day, so its first real run is during a live outage.
• Deploy tooling without a revert path. The pipeline can roll forward but has no first-class, one-command rollback action.

Diagnostic Workflow

Measure how fragile your recovery path is before the next bad deploy proves it.

1. Measure time-to-mitigate for deploy incidents. For past deploy-caused incidents, extract the time from “we decided to roll back” to “service recovered.” High or highly variable values indicate an improvised path.

2. Test the one-command claim. In staging, ask an engineer who did not write the service to roll it back to the previous version using only documentation. If they cannot do it quickly and confidently, you have no real runbook.

3. Audit migration reversibility. Review recent deploys for schema changes. Classify each as reversible (additive, expand/contract) or irreversible (destructive, non-additive). Irreversible migrations coupled to code are rollback landmines.

4. Inventory non-reverting state. For each service, list what a code rollback does NOT undo: flags, caches, queued jobs, data written in the new format. Every unlisted item is an incident surprise.

5. Check for a pinned known-good. Verify the release system exposes the previous healthy version as a single, unambiguous rollback target.

6. Rehearse and time it. Run a game-day rollback and record the elapsed time and any manual guesswork. That number is your realistic recovery floor.

Example Root Cause Analysis

A deploy to an orders service raised the error rate within two minutes; the alert and diagnosis were fast, and the team decided to roll back at the four-minute mark. Recovery, however, took 38 more minutes.

The timeline: the engineer knew the previous version number but the deploy tool had no one-command revert, so they hand-assembled a deploy command from a previous incident’s Slack thread. Then someone noticed the release had included a migration that dropped a now-unused column. Rolling the code back would leave the old code expecting that column — a crash risk. The team paused for 20 minutes, pulled in a database engineer, and manually recreated the column before daring to revert the code.

The diagnosis was correct in four minutes; the other 38 were spent because the rollback was improvised and coupled to an irreversible, destructive migration. Root cause of the slow MTTR was a missing, untested rollback runbook plus a non-additive migration shipped with code. The fixes: adopt expand/contract migrations so schema changes are always backward-compatible during a deploy window; add a first-class one-command rollback to the pipeline with a pinned known-good target; and rehearse the path in a quarterly game day.

Prevention Best Practices

• Write and version a rollback runbook. Document the exact trigger, the one command, expected duration per step, and the verification check — stored with the service, not in Slack.
• Make rollback one command. Give the deploy pipeline a first-class revert action with a pinned, unambiguous known-good target.
• Use expand/contract migrations. Ship only backward-compatible schema changes so code rollback is always safe within the deploy window; defer destructive cleanup to a later, separate release.
• Plan for non-reverting state. Explicitly design how flags, caches, and in-flight data are handled during a rollback, and put it in the runbook.
• Prefer flag kill-switches for gated features. When impact sits behind a feature flag, a flip mitigates in seconds without a deploy at all.
• Rehearse rollbacks in game days. Practice the path in staging so its first real execution is not during a live outage; record the realistic recovery time.
• Add safety gates. Require a pre-rollback compatibility check (current vs. target version, in-flight migrations) and a backup gate for any path that touches data.

Quick Reference

Signal	Healthy	Broken
Rollback trigger	one documented command	improvised from memory
Time-to-mitigate (deploy incident)	seconds to low minutes	tens of minutes, variable
Migration style	expand/contract, reversible	destructive, coupled to code
Known-good version	pinned, one-click	hunted for mid-incident
Non-reverting state	planned in runbook	discovered during outage
Rehearsal	game-day tested	first run is live

Conclusion

The cruelest MTTR anti-pattern is the one that strikes after you already know the fix. A manual, undocumented rollback turns “just roll it back” into a tense, risky improvisation exactly when calm and speed matter most. The remedy is preparation, not heroics: a versioned rollback runbook, a one-command revert with a pinned known-good, backward-compatible migrations, an explicit plan for state that does not revert, and rehearsal before the real thing. When rollback is boring and instant, the recovery phase — usually the biggest slice of MTTR — collapses to near zero. For the rollback-design and kill-switch prompts that help build these paths, see the Reduce MTTR category.