⏪ Postmortems for Failed Deploys: When the Rollback Doesn't Save You

A bad deploy is a routine event. You ship a change, something’s wrong, you roll back, you’re annoyed but fine. The incidents that actually hurt are the ones where you reached for the rollback and it didn’t save you — it failed to trigger, or it completed but restored a state that was already incompatible with a migration that had run, or it introduced a brand-new failure on the way down. Those are the deploys that turn a five-minute annoyance into an hour-long outage, and they share a treacherous feature in the postmortem: teams write them up as a single “bad deploy” story.

That single-story framing is the mistake. A failed deploy where rollback also failed is two incidents stacked on top of each other, with two different root causes living in two different places. Fix only the deploy and you’ve left an untrustworthy rollback armed for next time. The analysis has to split them, and that split is the structure AI can enforce when a tired team would blur it.

Two timelines, not one

The first move is to reconstruct two separate timelines with the rollback decision marked between them: the deploy degradation, and then the rollback attempt. The moment the team decided to roll back is the hinge, and separating before and after it keeps the two failures from smearing together into “everything went wrong around 3 a.m.”

Analyze this deploy-then-rollback incident as TWO problems.

1. Reconstruct two timelines: the deploy degradation, and the
   rollback attempt — clearly separated, with the moment the team
   decided to roll back marked between them.
2. DEPLOY FAILURE: what shipped, why the rollout strategy (or its
   absence) let it reach impact, which gate/check should have
   caught it.
3. ROLLBACK FAILURE (separate root cause): did rollback fail to
   trigger, fail to complete, restore a bad state, or cause new
   harm (incompatible schema, cached bad config)?
4. TRUST GAP: was rollback ASSUMED safe but never tested under
   these conditions?
5. Action items in TWO buckets: safer deploys (gates, canary,
   progressive rollout) and trustworthy rollback (tested rollback,
   forward-fix readiness, schema compatibility).

Rules: The rollback decision was made with the info available —
analyze the system that made rollback unsafe, not the person who
pulled the lever. Mark unconfirmed sequence details [UNVERIFIED].

Deploy details / rollback details / signals: <paste>

Why the rollback failure is its own root cause

The reason to treat the rollback as a separate analysis isn’t pedantry — it’s that the fixes live in completely different places. Safer deploys come from the forward path: gates that validate the change, canaries that catch it on 1% of traffic, progressive rollout that limits blast radius. None of those help your rollback. A trustworthy rollback comes from the reverse path: actually testing that the rollback works under realistic conditions, ensuring schema and data compatibility so rolling back code doesn’t collide with a migration, and having a forward-fix plan ready for the cases where rollback genuinely isn’t safe.

If your postmortem produces one bucket of action items, it almost always fills it with deploy-safety fixes — because that’s where the story started — and quietly leaves the rollback exactly as broken as it was. The two-bucket structure forces the question that matters: what made the rollback unsafe, and what would make it trustworthy?

The trust gap is usually the latent condition

Almost every failed rollback has the same latent condition underneath it: rollback was assumed to work and never tested for this scenario. Teams test their deploys constantly and their rollbacks approximately never, so the rollback path is the least-exercised code in the whole pipeline — right up until the worst possible moment, when it’s the only thing standing between you and an extended outage. After a human verifies the sequence, the analysis surfaces it cleanly:

Rollback failure analysis (distinct from the deploy failure)

The deploy included a database migration that added a NOT NULL column with application code that populated it. When the deploy degraded and the team rolled back the code at 03:14, the migration had already run. The rolled-back code didn’t know about the new column and its inserts failed the NOT NULL constraint, turning a partial degradation into a full write outage.

Root cause of the rollback failure: code rollback was assumed safe, but the deploy was not backward-compatible at the database layer — the migration and code were coupled, so rolling back one without the other left an inconsistent state. This path had never been tested.

Trust gap: the runbook said “roll back via the standard pipeline” and the team did exactly that. The system, not the responder, made rollback unsafe.

That’s two findings the single-story version would have collapsed into “the deploy went badly.” The deploy bucket gets a canary; the rollback bucket gets “enforce expand/contract migration patterns so code and schema can roll back independently, and add a test that exercises rollback against a migrated database.”

Don’t blame the lever-puller

The blameless framing has a specific shape in a deploy incident, because rollback decisions are made under pressure with partial data and they’re easy to second-guess in hindsight. “They should have forward-fixed instead of rolling back” is the classic one — and it’s blame wearing operational language, because in the moment, with what was known, rolling back was a reasonable call. The finding isn’t that the responder chose wrong; it’s that the system presented an unsafe rollback as a safe, standard option. The prompt keeps the analysis on why the rollback path was unsafe, and marks any uncertain sequencing as unverified so the two timelines aren’t stitched together from assumption.

The human owns the writeup

The model enforces the split, reconstructs both timelines, and produces two buckets of action items. A person still owns the final narrative and decides which fixes to fund — and notably, the rollback-trust work is the part most likely to get deprioritized precisely because rollbacks “usually work,” which is exactly the complacency that produced the incident. Resist that; the untested rollback is a loaded gun.

The failed-deploy prompt is in the prompts library, and it builds on the general incident postmortem work with the deploy-and-rollback structure these incidents specifically need. For the surrounding document, the blameless postmortem guide sets the framing.

When the rollback doesn’t save you, write up two incidents. Fix the deploy and the rollback — because the next bad deploy is coming, and you want the lever to actually work.