🔐 Vault Error: 'node not found' Removing or Joining a Raft Peer

Exact Error Message

$ vault operator raft remove-peer vault-2
Error removing peer: Error making API request.

URL: POST https://vault-0.vault-internal:8200/v1/sys/storage/raft/remove-peer
Code: 500. Errors:

* node not found

Joins fail with their own message:

$ vault operator raft join https://vault-0.vault-internal:8200
Error joining the node to the Raft cluster: Error making API request.

URL: POST https://vault-1.vault-internal:8200/v1/sys/storage/raft/join
Code: 500. Errors:

* failed to join raft cluster: failed to get raft challenge

What It Means

Vault’s Integrated Storage keeps its own membership list: a set of peers, each identified by a node_id string chosen in the server’s configuration file (or defaulting to the hostname). vault operator raft remove-peer looks up the ID you passed in that list and returns node not found when nothing matches. Almost always the ID you typed is not the ID the cluster knows — a pod was rebuilt with a different node_id, someone passed the DNS name instead of the ID, or the peer was already removed by autopilot’s dead server cleanup.

Join failures are the mirror image. A joining node contacts an existing member’s API address, receives an encrypted challenge it must decrypt with the cluster’s unseal keys, and only then is added as a peer. That handshake breaks when the joining node cannot reach the leader’s cluster_addr on port 8201, when it advertises an address the rest of the cluster cannot dial back, or when it starts with a non-empty data directory left over from a previous cluster. Membership and unsealing are also separate steps: a node that has joined but is still sealed will appear in the peer list without being a healthy voter.

Common Causes

• The node_id passed to remove-peer differs from the one in list-peers — often a hostname was used instead of the configured ID.
• A node was rebuilt with a new node_id while its old entry remained, leaving an orphaned peer that never returns.
• Autopilot already reaped the dead server, so a later manual remove-peer finds nothing.
• cluster_addr is unset or wrong, so peers advertise an unreachable address and the 8201 cluster port never connects.
• retry_join stanzas point at API addresses that are not resolvable from inside the joining container or VM.
• The node’s data directory was reused after a rebuild, so it carries stale Raft state from a different cluster and refuses to join cleanly.

Diagnostic Commands

Ask the cluster who it thinks its members are — this is the authoritative list of IDs:

vault operator raft list-peers

Check autopilot’s view, which shows health, voter status, and whether a node is considered failed:

vault operator raft autopilot state

Confirm which node you are actually talking to and whether it is the leader:

vault status
vault read -format=json sys/leader | jq '{leader_address, is_self, ha_enabled}'

Verify the joining node can reach the leader on both the API and cluster ports:

nc -vz vault-0.vault-internal 8200
nc -vz vault-0.vault-internal 8201

Inspect the running configuration for the node’s identity and advertised addresses:

grep -E 'node_id|cluster_addr|api_addr|retry_join' /etc/vault.d/vault.hcl

Watch the server log during a join attempt for the challenge exchange:

journalctl -u vault -n 100 --no-pager | grep -iE 'raft|join|challenge|peer'

Step-by-Step Resolution

1. Get the real peer IDs before doing anything destructive. The Node column is what remove-peer expects, and it is frequently not the DNS name you have been using:

vault operator raft list-peers

Node       Address                              State       Voter
----       -------                              -----       -----
vault-0    vault-0.vault-internal:8201          leader      true
vault-1    vault-1.vault-internal:8201          follower    true
node-7f2a  10.0.3.41:8201                       follower    false

2. Remove the peer using the exact ID from that output, and run the command against the current leader. Removing a peer is a write operation and will fail on a standby:

VAULT_ADDR=https://vault-0.vault-internal:8200 \
  vault operator raft remove-peer node-7f2a

3. Fix node_id uniqueness and stability so the orphan cannot recur. Each node needs an ID that survives restarts but is never reused by a different machine with different data:

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr = "https://vault-0.vault-internal:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-1.vault-internal:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-2.vault-internal:8200"
  }
}

cluster_addr = "https://vault-1.vault-internal:8201"
api_addr     = "https://vault-1.vault-internal:8200"

List every node in retry_join, including the node itself — Vault ignores its own entry, and a full list means any node can bootstrap regardless of which one is elected leader.

4. Clear stale state before rejoining a rebuilt node. A data directory carried over from a previous cluster is the usual cause of a join that fails the challenge. Confirm the node holds nothing you need, then start clean:

systemctl stop vault
mv /opt/vault/data /opt/vault/data.bak-$(date +%s)
mkdir -p /opt/vault/data && chown vault:vault /opt/vault/data
systemctl start vault

5. Join and then unseal. A joined node stays sealed and cannot serve requests or count as a healthy voter until it is unsealed with the cluster’s keys — auto-unseal handles this for you, but with Shamir keys it is a manual step:

VAULT_ADDR=https://vault-1.vault-internal:8200 \
  vault operator raft join https://vault-0.vault-internal:8200

VAULT_ADDR=https://vault-1.vault-internal:8200 vault operator unseal
VAULT_ADDR=https://vault-1.vault-internal:8200 vault status

If the node reports itself sealed after joining, Vault error: “Vault is sealed” covers the unseal path; if commands against it return a redirect or refuse writes, see Vault error: “Vault is in standby mode”.

6. Let autopilot manage membership going forward. Dead server cleanup removes failed peers automatically once they exceed the configured timeout, which prevents the orphaned-peer situation entirely:

vault operator raft autopilot state

vault operator raft autopilot set-config \
  -cleanup-dead-servers=true \
  -dead-server-last-contact-threshold=10m \
  -min-quorum=3 \
  -server-stabilization-time=30s

Keep -min-quorum at your intended voter count so autopilot never reaps a node down to a cluster that cannot elect a leader.

7. If you have already lost quorum — more than half the voters gone, no leader, writes failing — do not try to remove peers. Recover from a snapshot instead. Take snapshots routinely so this path is available:

vault operator raft snapshot save /backup/vault-$(date +%F).snap

vault operator raft snapshot restore -force /backup/vault-2026-07-18.snap

Restore onto a single unsealed node, verify it becomes leader with vault operator raft list-peers, then bring the remaining nodes up empty and let retry_join re-add them.

Prevention

• Set node_id explicitly in configuration and derive it from a stable identity (StatefulSet ordinal, instance name) rather than letting it default to a changing hostname.
• Always set cluster_addr and api_addr to addresses every other node can resolve and dial, and confirm 8201 is open between all peers.
• List all cluster members in retry_join so bootstrapping never depends on one specific node being up.
• Enable autopilot dead server cleanup with a min-quorum matching your voter count, so replaced nodes are reaped without manual remove-peer.
• Never reuse a Raft data directory across rebuilds; treat the directory as belonging to one node identity for its whole life.
• Schedule vault operator raft snapshot save on a timer and rehearse a restore, because snapshot recovery is the only real answer to quorum loss.

Related Errors

• failed to get raft challenge — the join handshake could not complete, usually a cluster-port or address problem.
• local node not active but active cluster node not found — no leader is elected, a symptom of quorum loss.
• Vault is sealed — the node is a member but cannot participate until unsealed.
• connection refused when checking seal status — the process is down or bound to a different address entirely.

Frequently Asked Questions

Why does remove-peer say the node is not found when I can see it in my inventory? Because Raft membership uses node_id, not DNS names or instance IDs. Run vault operator raft list-peers and copy the value from the Node column verbatim — that is the only identifier the API accepts.

Can I run remove-peer against any node? No. It is a write to the storage backend, so it must go to the active node. Point VAULT_ADDR at the leader, or you will get a standby redirect rather than a useful result. See Vault error: “connection refused” on seal status if you cannot reach it at all.

Does a joined node count toward quorum immediately? Not until it is unsealed and has finished stabilizing. Autopilot’s server-stabilization-time deliberately holds a new node as a non-voter until it has been healthy for that duration, which prevents a flapping node from destabilizing elections.

I lost two of three nodes — can I just remove them? Removing peers requires a leader, and with one node out of three you have none. Recover from a snapshot onto a single node, confirm it is the leader, then rejoin the replacements. More cluster and storage fixes are collected in the Vault guides.