🚨 Uncontrolled Changes During an Outage: When the Fix Makes It Worse

Overview

Some of the worst moments in an incident are self-inflicted. The original fault is understood and someone is mitigating it — and meanwhile three other responders, acting in good faith, are each making changes: one restarts a service, another rolls back a deploy, a third bumps a config value, and a fourth kicks off a “quick” scale-up. Now the system’s behavior is the sum of four uncoordinated interventions, nobody knows which change caused the latest symptom, and the team can no longer reason about cause and effect. The fix has become a second incident.

Uncontrolled concurrent change during an active incident is a process failure mode, not a technical one. It is driven by urgency, adrenaline, and the entirely reasonable instinct to do something. But in a live incident, undisciplined action destroys the one thing you need most: a clean signal linking each change to its effect. When multiple untracked changes land at once, rollback becomes ambiguous, the timeline becomes unreconstructable, and the postmortem cannot separate the original cause from the damage the response added.

The antidote is not to freeze — it is to make change coordinated and observable: one change at a time, announced, owned, and logged, with the incident commander controlling the pace.

Symptoms

• New, different symptoms appear shortly after responders start making changes — the failure mode shifts rather than resolves.
• Nobody can say what the current state is: which version is deployed, what config is live, which flags are on, how many replicas are running.
• Two changes conflict — a rollback and a config push, or a scale-up and a restart — and the system ends up in a state no one intended.
• The timeline in chat is a jumble of “I just…”, “wait, did you…”, “who changed…”, with overlapping actions and no timestamps of record.
• A mitigation can’t be cleanly reverted because other changes landed on top of it.
• The postmortem can’t attribute the recovery (or a secondary outage) to a specific action.
• Responders are stepping on each other — restarting a service someone else is mid-deploy on.

Common Root Causes

• No single change owner / no incident commander controlling pace. With no one gating actions, every responder acts independently and in parallel.
• Urgency bias — “just do something.” Under pressure, doing multiple things at once feels faster than the disciplined one-change-at-a-time approach.
• Poor communication of intent. Responders make changes without announcing them first, so others can’t account for them.
• No shared change log. Actions live in people’s heads and scattered chat lines, not in a single authoritative timeline.
• Everyone has write access and no coordination norm. Broad production access without a “declare before you touch” culture invites parallel intervention.
• Concurrent normal changes not frozen. A routine deploy pipeline, an automated job, or an unrelated team’s change lands during the incident because deploys weren’t halted.
• Speculative fixes. Trying several plausible fixes simultaneously to “see what works” — which guarantees you won’t know what worked.
• Automation acting concurrently. Autoscalers, self-healing controllers, or CI keep changing the system while humans also change it.

Diagnostic Workflow

When you suspect the response itself is causing confusion, the move is to regain control of change before continuing to debug.

1. Call a change freeze on the bridge. The IC states out loud: “Freeze — no changes to production without going through me.” This is the single most effective action.

   IC: "Change freeze in effect. If you want to make a change,
        state it here, wait for go, then make ONE change and report the result."

2. Halt concurrent automated and routine changes. Pause the deploy pipeline, disable relevant autoscaling/self-healing if it is fighting you, and stop any batch jobs so humans and automation aren’t both intervening.

   # Stop CI/CD from shipping mid-incident
   # Pause a Kubernetes deployment rollout so nothing new rolls
   kubectl rollout pause deployment/checkout

3. Reconstruct current state. Establish the ground truth: what is actually deployed, what config/flags are live, replica counts. Get everyone onto the same picture before touching anything.

   kubectl get deploy checkout -o jsonpath='{.spec.template.spec.containers[0].image}'
   kubectl get deploy checkout -o jsonpath='{.spec.replicas}'

4. Build the change log retroactively. Have each responder state, with timestamps, every change they made since the incident began. Get it into one authoritative timeline.

5. Impose one-change-at-a-time discipline. For each proposed action: announce it, get IC go, make exactly one change, observe the effect, log it, then decide the next. Never batch.

6. Identify and safely unwind harmful changes. With a clean log, determine which response changes made things worse and revert them one at a time, verifying after each.

7. Verify against a known-good state before considering the system stable, so you are reasoning from a controlled baseline rather than an accumulation of interventions.

Example Root Cause Analysis

Incident: A checkout service began erroring at 20:30. By 20:55 it was worse and behaving erratically, with intermittent 500s and timeouts that didn’t match the original signature.

Timeline reconstruction:

• 20:30 — A dependency slowed, causing checkout timeouts. Understood cause.
• 20:34 — Responder A rolled back the last checkout deploy (reasonable).
• 20:36 — Responder B, unaware, restarted the checkout pods (interrupting A’s rollout mid-flight).
• 20:38 — Responder C bumped a connection-pool config and pushed it.
• 20:41 — The autoscaler, reacting to the churn, scaled replicas up then down.
• 20:55 — The service was now a mix of old and new pods, a half-applied rollback, a new pool config, and a fluctuating replica count. Nobody could say what was live.

Contributing factors: (1) no IC gating change, so four interventions landed in parallel; (2) no announce-before-change norm — B restarted into A’s rollout; (3) deploy pipeline and autoscaler kept changing the system alongside humans; (4) no shared change log, so state was unknowable.

Corrective actions: require an IC to gate all production changes during declared incidents, adopt an “announce, get go, one change, observe, log” protocol, auto-freeze the deploy pipeline when an incident of a given severity is declared, and add a live change-log convention (or bot) to the incident channel.

Prevention Best Practices

• Establish an incident commander who gates change. During a declared incident, no production change happens without going through the IC. This alone prevents most concurrent-change chaos.
• Adopt “announce, go, one change, observe, log.” Every change is stated before it’s made, made singly, and its effect recorded before the next.
• Auto-freeze routine changes on declaration. Wire incident declaration to pause the deploy pipeline and hold unrelated changes until stand-down.
• Keep a live change log in the incident channel — ideally a scribe or a bot capturing each action with a timestamp — so the timeline is authoritative, not reconstructed.
• Decide about automation deliberately. Know when to pause autoscalers/self-healing so they aren’t fighting responders, and re-enable them intentionally.
• Prefer one hypothesis-driven change over parallel speculative fixes, so you always know what caused what.
• Train responders that discipline is speed. Rehearse in game days that coordinated single changes resolve incidents faster than a flurry of parallel ones.

Quick Reference

Situation	Do this
Multiple people making changes	IC calls a change freeze immediately
Want to make a change	Announce it, get IC go, make ONE, observe, log
Deploy pipeline still running	Pause it for the incident’s duration
Autoscaler/self-healing fighting you	Pause deliberately, re-enable intentionally
Don’t know current state	Reconstruct deployed version/config/replicas first
Change made things worse	Revert one at a time, verify after each
Tempted to try several fixes at once	Don’t — one hypothesis-driven change at a time

Conclusion

In an incident, the instinct to act fast is right, but acting in parallel and untracked is how a team turns one outage into two. Uncontrolled concurrent change destroys the causal signal you depend on to reason about the system, makes rollback ambiguous, and leaves a timeline no one can reconstruct. The discipline that fixes it is not hesitation — it is coordination: an incident commander gating change, a strict announce-go-one-change-observe-log rhythm, frozen routine deploys, and a single live change log. Slower, controlled change is almost always the faster path to a clean recovery, and it is the only path to a postmortem that can actually explain what happened.