🚨 Split-Brain During Failover: When Both Sides Think They're Primary

Overview

Split-brain is one of the most damaging failure modes in incident response: a failover meant to restore availability instead leaves two nodes both believing they are the primary, each accepting writes independently. The cluster is technically “up” — that is what makes it so dangerous — but every write to the wrong node is data that will have to be reconciled or thrown away later.

This failure mode shows up in database clusters (Postgres, MySQL, MongoDB, Redis), in distributed consensus systems, and in active-active application tiers backed by shared state. It is usually triggered not by the original outage but by the response to it: an automated failover, a manual promotion, or a network partition that fools each side into thinking the other is dead. The result is silent data divergence that is often discovered hours later, when reconciliation is expensive and some writes are already unrecoverable.

Treating split-brain as a first-class incident scenario — with clear detection, a bias toward availability loss over divergence, and a rehearsed recovery — is the difference between a short outage and a multi-day data-integrity project.

Symptoms

Split-brain rarely announces itself cleanly. Watch for these signals:

• Two nodes both reporting a primary/master/leader role at the same time (SHOW REPLICA STATUS clean on two hosts, two leaders in a Patroni/etcd cluster).
• Diverging row counts or checksums between nodes that are supposed to be replicas of each other.
• Application errors about duplicate keys, conflicting versions, or unexpected constraint violations after a failover.
• Writes that “disappear” — a value read back differs depending on which node served the read.
• Replication that will not re-establish because both sides have advanced past a common point (diverged WAL/binlog positions, conflicting oplog).
• A network partition alert immediately preceding the failover, or a failover that fired while the old primary was still alive and reachable by clients.
• Monitoring showing the “failed” node still serving traffic through a stale connection pool, VIP, or DNS entry.

Common Root Causes

• No fencing (STONITH) of the old primary. Failover promoted a new primary but never guaranteed the old one stopped accepting writes. Both stay writable.
• Network partition, not a dead node. The old primary was healthy but isolated from the quorum. It kept serving one set of clients while the new primary served another.
• Loss of quorum / even-numbered voters. A cluster with no majority (two-node clusters, or a 3-node cluster that lost its witness) cannot agree on a single leader, so two sub-groups each elect one.
• Automated failover with too-aggressive timeouts. A brief blip trips the promotion before the primary is truly gone, while clients still reach the old one.
• Manual promotion during a comms breakdown. An engineer promotes a replica because “the primary is down,” not realizing another responder is already recovering the original.
• Stale traffic routing. The VIP, load balancer, service discovery, or a client-side connection pool keeps pointing some traffic at the demoted node.
• Async replication + forced promotion. Promoting a replica that had not caught up guarantees divergence the moment the old primary’s un-replicated writes are exposed.

Diagnostic Workflow

The first goal is to stop the bleeding — stop concurrent writes — before you diagnose anything deeply. Every second of dual-write makes reconciliation worse.

1. Confirm dual-primary. On each candidate node, check its declared role and whether it is currently accepting writes. Two writable primaries = confirmed split-brain.

   # Postgres: is this node in recovery (replica) or not (primary)?
   psql -tAc "SELECT pg_is_in_recovery();"   # 'f' means it thinks it is primary
   # Patroni cluster view
   patronictl -c /etc/patroni.yml list

2. Freeze writes to the loser immediately. Decide which node is the authoritative survivor (usually the one with the most/most-important writes or the one quorum agrees on) and cut writes to the other — pull it from the load balancer/VIP, set it read-only, or fence it.

   # MySQL: make the wrong node read-only right now
   mysql -e "SET GLOBAL super_read_only = ON;"

3. Quantify divergence. Compare positions and data between the two nodes to understand how far they diverged and during what window.

   # Postgres WAL positions / timelines
   psql -tAc "SELECT pg_last_wal_replay_lsn();"
   # Row-level: checksum critical tables on both nodes
   psql -tAc "SELECT count(*), md5(string_agg(id::text, ',' ORDER BY id)) FROM orders;"

4. Find the divergence window. Correlate the failover timestamp with the partition/promotion event to bound which writes on the losing node are orphaned.

5. Verify routing is now single-primary. Confirm the VIP/DNS/service discovery and every client pool point only at the survivor; a lingering stale pool re-creates the split.

6. Preserve the loser. Do not wipe or auto-rejoin the diverged node yet — its un-replicated writes may need to be extracted and replayed into the survivor.

Example Root Cause Analysis

Incident: A payments service went read-error-heavy at 02:14. Automated failover promoted db-2 to primary at 02:15. By 08:00, support reported customers seeing orders vanish and reappear.

Timeline reconstruction:

• 02:13 — A top-of-rack switch flapped, isolating db-1 from the quorum witness and from db-2, but not from a subset of app servers in the same rack.
• 02:15 — The cluster manager, seeing db-1 unreachable from quorum, promoted db-2. It did not fence db-1 because fencing was configured but the fencing agent required the same network path that had partitioned.
• 02:15–07:50 — App servers in db-1’s rack kept writing to db-1 (still primary, still reachable to them). All other app servers wrote to db-2. Two divergent order histories accumulated.
• 07:50 — Network was repaired; replication refused to re-establish (diverged timelines), surfacing the split.

Contributing factors: (1) fencing depended on the very network path that failed, so it silently no-op’d; (2) some clients resolved the primary via a per-rack cache rather than the cluster VIP, so they never followed the promotion; (3) failover timeout was tuned for fast recovery with no partition-detection guard.

Corrective actions: move fencing to an independent control path (out-of-band/IPMI), route all clients through the cluster-managed endpoint only, add a “is the old primary really gone?” quorum check before promotion, and add a dual-primary alert that pages instantly.

Prevention Best Practices

• Always fence the old primary before promoting. Promotion without a guarantee the old node is dead (STONITH via an independent path) is the number-one cause of split-brain.
• Maintain an odd number of voters and a real quorum. Use a witness/arbiter so a partition always leaves at most one side with a majority. Never run automated failover on a two-node cluster.
• Prefer losing availability over losing consistency for stateful systems: when quorum is lost, the safe default is to stop accepting writes, not to elect a second leader.
• Route every client through a single cluster-managed endpoint. Eliminate stale VIPs, per-host DNS caches, and long-lived connection pools that can outlive a promotion.
• Guard automated failover with partition detection and conservative timeouts so a brief blip cannot trigger promotion while the primary is alive.
• Alert on dual-primary directly — a check that pages the instant two nodes both report primary is your fastest detection.
• Rehearse failover in game days, including the partition scenario, so responders promote through the cluster manager rather than manually.

Quick Reference

Situation	Do this
Two nodes both report primary	Confirmed split-brain — stop writes to one now
Choosing the survivor	Keep the node quorum agrees on / with the most critical writes
Cutting off the loser	super_read_only/read-only + remove from VIP + fence
Diverged replication won’t rejoin	Do NOT wipe the loser; extract its orphaned writes first
Partition detected	Do not auto-promote; verify old primary is truly gone
Two-node cluster	Never auto-failover; require manual, fenced promotion
Post-recovery	Reconcile orphaned writes, then rebuild the loser from the survivor

Conclusion

Split-brain is a response-induced failure: the outage is what starts it, but an unfenced promotion, a lost quorum, or stale routing is what turns a short blip into diverging, corrupted data. The winning instinct during a failover incident is counterintuitive — when you cannot prove there is exactly one primary, stop writes rather than start a second one. Build fencing on an independent path, keep a real quorum, route all clients through one managed endpoint, and alert on dual-primary directly. Do that, and a failover stays a brief availability event instead of a multi-day data-integrity cleanup.