🧵 Parallelizing Incident Investigation With AI: Divide and Conquer

Four engineers on a SEV1 bridge, and three of them were watching the fourth type. That’s the dirty secret of “all hands on the incident”: more responders rarely means faster, because investigation tends to collapse into one person driving while everyone else spectates and occasionally suggests things that pull the driver off-track. We had four brains available and we were using them like one brain plus a peanut gallery. Investigation that should parallelize — checking the network, the database, the recent deploy, and the upstream all at once — was running serially, and our MTTR showed it.

The bottleneck isn’t headcount. It’s that nobody carved the incident into independent threads people could own without colliding. That carving is fast, structured work AI does well, and it’s where a small team turns into actual parallelism.

Why investigation collapses to serial

Two forces push every incident toward one-driver mode. First, coordination cost: if two people poke the same subsystem they trip over each other, so the safe default is “let one person go.” Second, shared anchoring: everyone’s watching the same screen, so everyone forms the same hypothesis and chases the same thread. The result is a serial walk through the hypothesis space when you have the people to walk several paths at once. Across the MTTR funnel, diagnosis is the fattest slice — and it’s the one that benefits most from being parallelized properly.

The unlock is a clean division of labor: independent threads, each with a clear owner, a clear scope, and a clear “report back” so the threads recombine. AI is good at proposing that division because it can take the symptom set and the candidate causes and partition them into non-overlapping investigation lanes.

Ask AI to partition the investigation

The prompt is about independence — threads that don’t require each other’s results to make progress.

You are coordinating a multi-person incident investigation. Given the symptoms and the ranked candidate causes, split the investigation into 3–4 independent threads that different responders can work in parallel without colliding. For each thread: a clear scope (what subsystem/hypothesis it covers), the first 2–3 checks (commands/queries), what result would confirm or kill the thread, and what not to touch (to avoid overlap with other threads). Flag any dependency between threads. Do not assign people. Do not recommend fixes — produce only the parallel investigation plan.

The output gives a small team something to fan out on:

Thread A — Recent deploy: Scope v2.41.0 rollout. Checks: kubectl rollout history, diff config between revisions, correlate error onset to deploy time. Kill if errors predate deploy. Don’t touch: upstream services. Thread B — Data layer: Scope DB + cache. Checks: pg_stat_activity count, slow-query log, cache hit ratio. Kill if all nominal. Don’t touch: app deploys. Thread C — Network/region: Scope us-east-1 isolation. Checks: per-region error split, LB health, cross-AZ latency. Kill if other regions also failing. Don’t touch: the database. Thread D — Upstream deps: Scope auth + payments-gateway. Checks: dependency error/latency dashboards, retry rates. Kill if all upstreams green. Dependency: If Thread A confirms a config change to DB pooling, Threads A and B must sync.

Now four responders each own a lane. They’re not watching one screen; they’re running their own checks and reporting kills. The incident commander assigns the humans to the threads — the model deliberately doesn’t, because that’s a people decision.

Threads report back as kills, not essays

Parallelism only pays off if recombination is cheap. The discipline: each thread reports a result, ideally a kill, in one line — not a narrative. The incident channel fills with falsifiable outcomes:

# Thread A owner runs and reports:
kubectl rollout history deploy/payments -n payments | tail -4
# -> "Thread A: errors started 01:52, deploy was 01:51. NOT killed, strong lead."

# Thread B owner runs and reports:
kubectl exec -n payments deploy/payments -- psql -tc \
  "select count(*) from pg_stat_activity;"
# -> "Thread B: 40/200 connections, slow-query log clean. KILLED."

# Thread C owner runs and reports:
sum by (region) (rate(payment_errors_total[5m]))
# -> "Thread C: errors in us-east-1 AND eu-west-1. Region-isolation theory KILLED."

In a few minutes you’ve killed two of four threads and narrowed a SEV1 to “recent deploy” with hard evidence, using four people genuinely in parallel instead of three watching one. That’s the MTTR win: wall-clock time compressed by doing independent work at the same time.

The AI plans the split; humans run and verify the threads

The model partitions the space and proposes checks. It does not run anything, assign anyone, or decide what’s true. Every thread’s kill is confirmed by a human running the check and reading the output — verify-first applies per-lane. This matters because a bad partition (two threads that secretly overlap, or a thread scoped around a wrong assumption) is caught by the humans working it, not taken on faith.

The incident commander’s role gets more important with parallelism, not less:

• Own the recombination. The threads produce kills; someone has to assemble them into “here’s what’s left.” That’s a human synthesis job. Pair it with an AI scribe keeping the kills in a live timeline.
• Enforce the no-touch boundaries. The whole point is non-collision. If Thread C’s owner starts poking the database, the parallelism breaks and you’re back to stepping on each other.
• Re-partition when a thread surprises you. If Thread A turns up something that reshapes the whole picture, regenerate the split rather than forcing the old lanes.

A couple of failure modes to watch:

• Too many threads. Four responders, four threads. Don’t let the model propose eight; threads without owners are just an unread to-do list.
• Fake independence. If two threads actually depend on the same unknown, the model should flag it — and you should sequence them, not run them in parallel and get confused.

You can prototype the split on the free incident assistant: paste symptoms and candidate causes and ask for an independent-thread plan, then notice how much more a small team can cover when nobody’s spectating. The prompt library has the partition prompt with the no-touch and dependency-flagging rules included.

Throwing people at an incident doesn’t help if they all watch one person type. The win is carving the problem into lanes people can own in parallel — and AI is fast at carving. The humans run the lanes, verify the kills, and recombine. That’s how a four-person bridge actually works four times as fast instead of one time as fast with an audience.