Rollback vs Roll-Forward: Deciding Under Pressure

A deploy goes out, something breaks, and now you’re standing at a fork with customers watching: do you roll back to the last known-good version, or roll forward with a fix? It’s one of the most common high-stakes decisions in incident response, it has to be made in minutes, and both wrong answers hurt. Roll back when you can’t and you might make it worse; roll forward when you should have reverted and you gamble customer-facing time on a fix that might not work.

I’ve made this call many times, gotten it wrong a few, and settled on a default that’s right far more often than not — plus a short set of questions that tell you when to deviate from it. Here’s the framework.

The default: roll back

When a recent change caused an incident, rolling back is the right default, and you should need a specific reason not to. The logic is simple: a rollback returns you to a state you know worked. It’s the fastest path to a known-good outcome, and its result is predictable. Rolling forward, by contrast, ships a new, untested-in-this-situation change into a live outage — you’re doing an emergency deploy of code that’s never run in production, under maximum stress, hoping it both fixes the problem and introduces no new one. That’s a gamble, and the middle of an outage is the worst time to gamble.

So the mental default is: if a change caused it and the change is revertible, roll back now, diagnose later. Stop the bleeding with the known-good version, then figure out the real fix calmly once customers are okay. You make the actual repair on your own schedule instead of the outage’s.

When rolling back doesn’t work

The default has real exceptions, and recognizing them fast is the skill. Roll forward instead when:

ROLL FORWARD (not back) WHEN:

[ ] The rollback is unsafe — a DB migration or schema change
    already ran and the old code can't read the new data.
    Reverting could cause data loss or corruption.
[ ] There's nothing to roll back to — the problem isn't from a
    recent deploy (bad data, external dependency, capacity).
[ ] Rolling back reintroduces a WORSE problem you just fixed
    (e.g. the deploy patched a security hole or data bug).
[ ] The fix forward is genuinely trivial and certain — a
    one-line, obviously-correct change (a bad env var, a typo
    in config) that's faster and safer than a full revert.
[ ] Rollback is technically impossible right now (stateful
    system, can't downgrade in place).

The migration case is the big one and the most dangerous. If your deploy included an irreversible schema or data migration, a naive rollback of the code against the already-migrated data can be catastrophic — the old code doesn’t understand the new schema, and now you have a data problem on top of an outage. This is why “can I even safely roll back?” must be one of the first questions, not an assumption.

The questions to ask, fast

When you hit the fork, run these in order. They resolve most decisions in under a minute:

ROLLBACK vs ROLL-FORWARD — DECISION QUESTIONS

1. Did a recent change cause this?
   No  → rollback probably won't help; mitigate/fix forward.
   Yes → continue.

2. Can I safely roll back? (Any migrations / stateful /
   irreversible data changes in that deploy?)
   No/unsure → do NOT blind-rollback; fix forward carefully
               or find a safe mitigation.
   Yes → continue.

3. Is a fix-forward trivial AND certain?
   Yes → fixing forward may be faster and fine.
   No  → ROLL BACK. Don't gamble an outage on an untested fix.

4. How confident am I in the diagnosis?
   Low → roll back (known-good beats a guess).
   High + trivial fix → fix forward is defensible.

Question 3 is where people talk themselves into trouble. “I’m pretty sure I know the fix, let me just push it” feels faster than a rollback and often isn’t — because “pretty sure” fixes fail, and now you’ve burned time and still have to roll back. Unless the fix is both trivial and near-certain, the known-good rollback wins.

Prefer reversible, and change one thing

Whichever way you go, two rules from general incident hygiene apply. First, prefer the more reversible option — if the rollback is clean and the fix-forward is a gamble, that asymmetry alone usually decides it. Second, change one thing at a time and announce it: “I’m rolling back the payments deploy now” lets the room catch “wait, that deploy included the migration” before you make it worse. The decision isn’t just yours to make silently; say it out loud so the people who know about the migration can stop you.

After the decision

Once you’ve acted, confirm it actually helped and held before declaring victory — a rollback that appears to work but relapses two minutes later means the deploy wasn’t the whole story. And regardless of which path you took under pressure, the calm, correct fix usually happens afterward: even if you rolled forward with a quick patch to stop the bleeding, you’ll often follow up with a properly-tested change once the incident’s over. The in-incident decision optimizes for stopping impact fast and safely; the real engineering happens after.

Common mistakes

Fixing forward on “pretty sure.” Gambling customer time on an untested, un-certain fix when a known-good rollback was right there. Roll back unless the fix is trivial and certain.
Blind-rolling-back through a migration. Reverting code against already-migrated data and creating a data-loss incident. Always ask “can I safely roll back?” first.
Assuming a recent deploy caused it. Rolling back a change that had nothing to do with the problem, wasting time and adding churn. Confirm the change actually correlates.
Silent action. Rolling back or forward without announcing it, so the person who knew about the schema change couldn’t warn you in time.
Declaring done too early. Not confirming the choice held before standing down.

Where AI helps the call

The rollback/roll-forward decision benefits from fast, structured reasoning about risk — which is where a model assists, not decides. Paste what the deploy contained (including whether it had migrations), the symptoms, and the timing, and ask it to reason through rollback safety and the reversibility trade-offs: “given this deploy included a schema migration, what are the risks of rolling back the code, and what safer options exist?” It’s a useful second opinion for surfacing the migration-safety angle you might miss under pressure. The human owns the decision and the action — especially anything touching data. The Incident Response tool can structure this risk assessment from the incident details.

Wrapping up

At the rollback-versus-roll-forward fork, default to rolling back to known-good — it’s fast, predictable, and safe — and deviate only for specific reasons: an unsafe migration, nothing to revert to, or a fix that’s genuinely trivial and certain. Run the questions in order, prefer the reversible path, announce what you’re doing so someone can catch a hidden trap, and confirm it held. Save the properly-engineered fix for after the bleeding stops. Made this way, the call is far less agonizing than it feels in the moment.

AI risk assessments are advisory drafts. A human owns the rollback/roll-forward decision and every production action.