🐰 RabbitMQ Error: 'khepri_db' Metadata Store Enable and Replication Failure

Exact Error Message

Error:
enabling feature flag "khepri_db" failed:
{error,{khepri,timeout,
  <<"Khepri metadata store failed to initialize: timed out waiting for a leader">>}}

# In the node log:
[error] <0.812.0> Feature flags: `khepri_db`: failed to enable: {error,timeout}
[warning] <0.799.0> Ra: leader election for cluster rabbitmq_metadata took longer than expected
[error] <0.812.0> Khepri: timed out waiting for leader in cluster 'rabbitmq_metadata'

You may also see feature flag state is 'state_changing' when a previous enable attempt was interrupted, or {error,{node_not_running,...}} when a cluster member is offline.

What It Means

Khepri is RabbitMQ’s Raft-based metadata store (the replacement for Mnesia), backed by the Ra consensus library. It keeps virtual hosts, users, queues, exchanges, and bindings as replicated state on a Raft log. Enabling the khepri_db feature flag migrates all metadata from Mnesia into a Khepri Raft cluster and requires a healthy leader to be elected across the RabbitMQ nodes.

The error means the migration or the Khepri cluster could not elect a leader in time. Because Raft needs a majority (quorum) of members reachable and in agreement, any node being down, a network partition, or clock/connectivity issues between nodes will stall leader election and fail the enable operation. The feature flag can be left in a state_changing state, which blocks a clean retry until it settles.

Common Causes

• Not every cluster node is running when you enable khepri_db; the feature flag must be enabled cluster-wide with all members online.
• A network partition or blocked inter-node port (25672, plus the Erlang distribution/EPMD port 4369) prevents Ra members from reaching quorum.
• A previous enable attempt was interrupted, leaving the flag in state_changing.
• Mixed RabbitMQ versions in the cluster where some nodes don’t support khepri_db.
• Slow disks or an overloaded node so the Ra log/leader election exceeds the internal timeout.
• Firewall or security-group changes that severed the cluster mesh mid-migration.

Diagnostic Commands

Check the current state of the feature flag on the node you ran the command from:

rabbitmqctl list_feature_flags name state | grep khepri_db

Confirm which metadata store the node is actually using and whether it initialized:

rabbitmq-diagnostics metadata_store_status
rabbitmq-diagnostics status | grep -iE 'metadata|khepri'

Verify the cluster is fully formed and no partition is present before retrying:

rabbitmqctl cluster_status

Inspect the log for Ra/Khepri leader-election detail:

grep -iE 'khepri|ra: |leader|rabbitmq_metadata' \
  /var/log/rabbitmq/rabbit@$(hostname -s).log | tail -n 40

Step-by-Step Resolution

1. Confirm every node is up. Khepri must be enabled with all members online:

rabbitmqctl cluster_status

Look for all nodes under Running Nodes and an empty Network Partitions section.

2. If any node is down, start it and wait until it fully rejoins before doing anything else:

systemctl start rabbitmq-server
rabbitmqctl await_online_nodes 3   # replace 3 with your node count

3. Verify inter-node connectivity so Ra can reach quorum. From each node, confirm the distribution and clustering ports are reachable to peers:

rabbitmq-diagnostics ping
ss -ltnp | grep -E ':25672|:4369'

4. Re-run the enable once the cluster is healthy. Enabling on one node enables it cluster-wide:

rabbitmqctl enable_feature_flag khepri_db

5. If the flag is stuck in state_changing, give the internal retries time to settle, then re-check and re-issue the enable. Do not restart nodes mid-transition unless a node is genuinely hung:

rabbitmqctl list_feature_flags name state | grep khepri_db
rabbitmqctl enable_feature_flag khepri_db

6. Confirm success. The flag should read enabled and the metadata store should report Khepri as initialized:

rabbitmqctl list_feature_flags name state | grep khepri_db
rabbitmq-diagnostics metadata_store_status

khepri_db	enabled

Prevention

• Only enable khepri_db when the entire cluster is running and free of network partitions; check cluster_status first.
• Upgrade every node to a version that supports Khepri before enabling the flag; never enable it on a mixed-version cluster.
• Ensure the Erlang distribution port (25672) and EPMD (4369) are open between all nodes in firewalls and security groups.
• Enable the flag during a maintenance window so a slow leader election under load doesn’t time out.
• Export definitions beforehand so metadata can be restored if a migration needs to be redone.
• Watch disk latency on the data volume; Raft log writes are sensitive to slow storage.

Related Errors

• BOOT FAILED ... schema_integrity_check_failed — a Mnesia-side schema problem rather than a Khepri leader-election failure.
• Network partition detected — the underlying partition that also blocks Khepri quorum.
• {error,{node_not_running,...}} — a cluster member is offline when a cluster-wide operation runs.
• feature flag state is 'state_changing' — an interrupted enable that must settle before you retry.

Frequently Asked Questions

Do I have to enable khepri_db on every node separately? No. Feature flags are cluster-wide, so enabling it on one node enables it across the cluster, but every node must be online and reachable for the operation to succeed.

Can I roll back to Mnesia after enabling Khepri? No. Enabling khepri_db is a one-way migration, so export your definitions first and treat the change as permanent.

Why does leader election time out even though the nodes are up? Raft needs to reach a majority over the inter-node ports; a blocked 25672/4369, a partition, or very slow disks will stall election even when the processes are running.

How do I generate the exact recovery steps for my cluster? Describe your node count and error in the DevOps AI prompt library to get tailored diagnostics, or browse more RabbitMQ guides.