🚨 OpenStack Error Guide: 'WSREP has not yet prepared node' — Recover the Galera Cluster

Overview

WSREP has not yet prepared this node for application use is the error MariaDB/Galera returns when a node is up and accepting connections but is not in the Primary Component of the cluster. Galera refuses reads and writes on a node that isn’t part of a quorum, to avoid serving stale or divergent data. Because every OpenStack service stores its state in this database (Keystone tokens, Nova instances, Neutron ports, Cinder volumes), a non-primary Galera cluster takes the whole control plane down at once.

The literal errors you will see:

ERROR 1047 (08S01): WSREP has not yet prepared this node for application use

(pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared this node for application use')
2026-07-08 03:11:02 [ERROR] WSREP: failed to open gcomm backend connection: 110: ... (FATAL)
2026-07-08 03:11:02 [Note] WSREP: view(view_id(NON_PRIM, ...))

It appears whenever a service issues SQL — Keystone validating a token, Nova reading an instance, an openstack command, or Horizon loading a panel. The signature, like a message-bus outage, is that failures hit every service simultaneously and point at the database.

Symptoms

• Every OpenStack service logs DB errors at once; openstack commands fail with 500s.
• Direct SQL returns ERROR 1047 ... WSREP has not yet prepared this node.
• Services log pymysql.err.OperationalError (1047, ...) or “Can’t connect to MySQL server / lost connection”.
• HAProxy shows the MariaDB backend down or flapping.

docker exec mariadb mysql -uroot -p"$DB_ROOT" -e "SHOW STATUS LIKE 'wsrep_cluster_status';" 2>/dev/null

+----------------------+-----------+
| Variable_name        | Value     |
+----------------------+-----------+
| wsrep_cluster_status | non-Primary |
+----------------------+-----------+

Common Root Causes

1. Lost quorum (non-Primary component)

Enough nodes dropped that the survivors can’t form a majority, so they go non-Primary and stop serving. A 3-node cluster losing 2 nodes is the classic case.

docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_ready');" 2>/dev/null

| wsrep_cluster_size   | 1           |
| wsrep_cluster_status | non-Primary |
| wsrep_ready          | OFF         |

wsrep_ready = OFF with a shrunken cluster_size = lost quorum.

2. Full cluster shutdown / cold start with no bootstrap

After a power event or docker restart of all DB nodes, no node was told it holds the most advanced state, so none becomes Primary. The cluster waits forever.

docker ps --filter name=mariadb --format '{{.Names}} {{.Status}}'
docker logs mariadb 2>&1 | grep -iE 'grastate|safe_to_bootstrap|NON_PRIM|gcomm' | tail

WSREP: waiting for gcomm to become primary ...

3. Network partition (split-brain risk)

An inter-node network blip partitions the cluster; the minority side(s) go non-Primary. Galera’s own quorum logic prevents true split-brain, but the minority stops serving.

docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS LIKE 'wsrep_incoming_addresses';" 2>/dev/null
docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS LIKE 'wsrep_cluster_conf_id';" 2>/dev/null

Fewer wsrep_incoming_addresses than expected nodes = a partition.

4. SST failure on a joining node

A node rejoining via State Snapshot Transfer (mariabackup/rsync) failed — bad credentials, disk full, or version mismatch — so it never reaches Synced.

docker logs mariadb 2>&1 | grep -iE 'SST|mariabackup|rsync|Donor|Joiner' | tail -15
docker exec mariadb mysql -uroot -p"$DB_ROOT" -e "SHOW STATUS LIKE 'wsrep_local_state_comment';" 2>/dev/null

| wsrep_local_state_comment | Joining: receiving State Transfer |   # stuck here

5. Wrong bootstrap node (stale data)

A node with safe_to_bootstrap: 0 (or a lower seqno) was bootstrapped, so the cluster came up on stale data or refused.

docker exec mariadb cat /var/lib/mysql/grastate.dat

seqno: 42
safe_to_bootstrap: 0     # this node must NOT be the bootstrap node

6. Disk full / GCache or tmp exhaustion

A full data disk stalls Galera and drops the node out of the Primary component.

df -h /var/lib/docker /var/lib/mysql 2>/dev/null
docker exec mariadb df -h /var/lib/mysql

Diagnostic Workflow

Step 1: Confirm it’s the database, cluster-wide

openstack token issue 2>&1 | tail -3      # fails with a 500/DB error
docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS WHERE Variable_name IN ('wsrep_ready','wsrep_cluster_status','wsrep_cluster_size','wsrep_local_state_comment');" 2>/dev/null

wsrep_ready=OFF / non-Primary confirms the cluster, not one service, is the problem.

Step 2: Check every DB node’s state

# On each controller
for h in controller-01 controller-02 controller-03; do
  echo "== $h =="; ssh "$h" 'docker exec mariadb mysql -uroot -p"$DB_ROOT" \
    -e "SHOW STATUS LIKE '\''wsrep_local_state_comment'\'';" 2>/dev/null'
done

Look for which nodes are Synced, Joining, or down.

Step 3: Find the most advanced node

The node with the highest seqno in grastate.dat holds the newest data and should seed recovery.

docker exec mariadb cat /var/lib/mysql/grastate.dat
# If mysqld is stopped, recover the position:
docker exec mariadb mysqld --wsrep-recover 2>&1 | grep -i 'Recovered position'

Step 4: Read the Galera log for the failure mode

docker logs mariadb 2>&1 | grep -iE 'WSREP|SST|NON_PRIM|gcomm|grastate|safe_to_bootstrap' | tail -30

SST failures, NON_PRIM, and “waiting for gcomm to become primary” each point at a different fix.

Step 5: Recover — bootstrap from the most advanced node

For a full-cluster cold start, on only the highest-seqno node, mark it safe and bootstrap, then start the others so they SST from it. With Kolla-Ansible, prefer the built-in recovery which does this for you:

# Kolla-Ansible (recommended) — pick the correct bootstrap node automatically
kolla-ansible mariadb_recovery

# Manual, only if you must and you have identified the seed node:
#   set safe_to_bootstrap: 1 in grastate.dat on the seed node ONLY, then bootstrap it,
#   then start the remaining nodes so they resync via SST.

Step 6: Verify quorum and let services recover

docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS WHERE Variable_name IN ('wsrep_cluster_size','wsrep_cluster_status','wsrep_ready');" 2>/dev/null
openstack token issue -c id           # control plane responds again

Example Root Cause Analysis

After a rack power blip, every OpenStack API returns 500 and Keystone logs:

(pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared this node for application use')

All three MariaDB containers are Up, but each reports non-Primary:

docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS LIKE 'wsrep_cluster_status';" 2>/dev/null

| wsrep_cluster_status | non-Primary |

The Galera log shows the tell for a cold start:

WSREP: waiting for gcomm to become primary

Every node came back at once and none was designated to bootstrap, so no Primary Component ever formed. Comparing recovered positions finds the most advanced node:

for h in controller-01 controller-02 controller-03; do
  echo "== $h =="; ssh "$h" 'docker exec mariadb cat /var/lib/mysql/grastate.dat | grep seqno'
done

== controller-01 == seqno: 91422
== controller-02 == seqno: 91422
== controller-03 == seqno: 91410

controller-01 (highest seqno) is the correct seed. Rather than hand-edit grastate.dat, run Kolla’s recovery, which bootstraps the right node and resyncs the rest:

kolla-ansible mariadb_recovery
docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS LIKE 'wsrep_cluster_size';" 2>/dev/null   # -> 3
openstack token issue -c id                              # APIs recover

Root cause: a full-cluster shutdown with no automatic bootstrap left every node non-Primary. Bootstrapping from the most-advanced node restored quorum. Bootstrapping the wrong (lower-seqno) node would have discarded committed transactions.

Prevention Best Practices

• Run an odd node count (3 or 5) so the cluster can lose a minority and keep quorum.
• Never docker restart all DB nodes at once; if you must cold-start, always bootstrap from the highest-seqno node — use kolla-ansible mariadb_recovery rather than editing grastate.dat by hand.
• Monitor wsrep_cluster_size, wsrep_cluster_status, wsrep_ready, and wsrep_local_state_comment; page when size drops or status leaves Primary.
• Alert on data-disk usage for /var/lib/mysql; a full disk drops nodes out of the cluster.
• Keep SST credentials and mariabackup versions consistent across nodes so a rejoin can’t fail on a version/auth mismatch.
• Sync clocks and pre-open the Galera ports (3306, 4567/tcp+udp, 4568, 4444) between DB nodes; a partition here causes non-Primary.
• Take verified Galera backups (mariabackup) so a bad bootstrap is recoverable.
• For triage of a cluster-wide DB outage, paste the wsrep_* status and Galera log into the free incident assistant, and see more OpenStack guides.

Quick Command Reference

# Is the whole DB non-Primary?
docker exec mariadb mysql -uroot -p"$DB_ROOT" -e \
  "SHOW STATUS WHERE Variable_name IN ('wsrep_ready','wsrep_cluster_status','wsrep_cluster_size','wsrep_local_state_comment');"

# Per-node state
docker exec mariadb mysql -uroot -p"$DB_ROOT" -e "SHOW STATUS LIKE 'wsrep_local_state_comment';"

# Find the most advanced node (highest seqno)
docker exec mariadb cat /var/lib/mysql/grastate.dat
docker exec mariadb mysqld --wsrep-recover 2>&1 | grep -i 'Recovered position'

# Read the Galera log
docker logs mariadb 2>&1 | grep -iE 'WSREP|SST|NON_PRIM|gcomm|safe_to_bootstrap' | tail -30

# Recover (Kolla-Ansible picks & bootstraps the right node)
kolla-ansible mariadb_recovery

# Verify quorum and control plane
docker exec mariadb mysql -uroot -p"$DB_ROOT" -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
openstack token issue -c id

Conclusion

WSREP has not yet prepared this node for application use means the MariaDB/Galera node is running but not in the Primary Component, so it refuses to serve — and because every OpenStack service depends on this database, the whole control plane goes down together. Typical root causes:

1. Lost quorum — too many nodes dropped to keep a majority.
2. A full cold start with no node designated to bootstrap.
3. A network partition pushing the minority non-Primary.
4. A failed SST leaving a joiner stuck.
5. Bootstrapping the wrong (stale, lower-seqno) node.
6. A full data disk dropping nodes out.

Check wsrep_cluster_status/wsrep_ready to confirm it’s cluster-wide, identify the highest-seqno node, and recover from that node — with Kolla-Ansible, kolla-ansible mariadb_recovery does this safely. The one rule that prevents data loss: always bootstrap from the most advanced node, never a stale one.