🔐 Vault Error: 'failed to take snapshot' Raft Integrated Storage Snapshot and Restore Failures

Exact Error Message

$ vault operator raft snapshot save /backup/vault-2026-07-19.snap
Error taking the snapshot: Error making API request.

URL: GET https://vault.internal:8200/v1/sys/storage/raft/snapshot
Code: 500. Errors:

* failed to take snapshot: failed to create snapshot: write /opt/vault/data/snapshots/tmp-snap-4471: no space left on device

On restore you may instead see:

$ vault operator raft snapshot restore /backup/vault-2026-07-19.snap
Error installing the snapshot: Error making API request.

URL: POST https://vault.internal:8200/v1/sys/storage/raft/snapshot
Code: 400. Errors:

* could not verify hash file, possibly the snapshot is using a different set of unseal keys; use the snapshot-force API to bypass this check

Or, when the node you targeted is not the leader:

Code: 500. Errors:

* local node not active but active cluster node not found

What It Means

Vault’s integrated storage (raft) keeps the entire encrypted data store in a local BoltDB file — raft.db — replicated across cluster peers by the Raft consensus protocol. A snapshot is a point-in-time, consistent copy of that FSM plus the Raft configuration, streamed out over sys/storage/raft/snapshot. Taking one is not free: Vault materialises the snapshot on the leader’s local disk before streaming it to the client, so a snapshot of a 12 GB store needs roughly that much free space in addition to the live raft.db. The most common 500 on snapshot save is simply the volume filling up mid-write.

Restore is a different beast. The snapshot contains data encrypted with the barrier keys that were in effect when it was taken. When you restore, Vault verifies that the current cluster’s seal can actually decrypt the snapshot’s contents by checking a hash of the seal-wrapped material. If you restore a snapshot into a cluster initialised with different unseal keys — or a cluster using a different auto-unseal KMS key — that check fails and Vault refuses. The -force flag exists to bypass exactly this check, which is why it is dangerous: it will happily install a snapshot the running seal cannot decrypt, leaving you with a cluster full of unreadable ciphertext and no rollback.

Common Causes

• The data volume holding /opt/vault/data is full, so the leader cannot stage the snapshot file.
• The token used lacks a policy granting read on sys/storage/raft/snapshot (save) or create/update (restore).
• The snapshot was taken from a cluster with different unseal keys or a different auto-unseal key, so the hash verification fails on restore.
• The command targeted a standby or performance-standby node rather than the active node, and no active node could be resolved.
• raft.db has grown far beyond the logical data size because snapshot_threshold and trailing_logs are tuned too high, or dead peers are blocking log truncation.
• A failed or removed node still appears in the Raft configuration, so autopilot keeps the cluster in a degraded state and log compaction stalls.

Diagnostic Commands

Start by confirming the cluster’s peer set and who the leader is. Snapshot operations are served by the active node:

vault operator raft list-peers
vault status

Check autopilot’s view of health — this is where you find nodes that are unhealthy, non-voters, or stale:

vault operator raft autopilot state
vault operator raft autopilot get-config

Look at actual disk consumption on the storage path. Compare the logical size of raft.db against free space:

df -h /opt/vault/data
du -sh /opt/vault/data/raft/raft.db
ls -lh /opt/vault/data/raft/snapshots/

Confirm the token you are using actually has snapshot permission, rather than guessing:

vault token capabilities "$(vault print token)" sys/storage/raft/snapshot

You can also drive the endpoint directly with curl to see the raw HTTP status, which is useful when the CLI masks the underlying error:

curl -sS -o /tmp/vault.snap -w '%{http_code}\n' \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  https://vault.internal:8200/v1/sys/storage/raft/snapshot

If that curl fails with a certificate error, do not reach for -k or VAULT_SKIP_VERIFY as a fix. You may set VAULT_SKIP_VERIFY=true for a single command purely to prove the failure is TLS trust and not authorization — then immediately restore proper CA trust by pointing VAULT_CACERT at the issuing CA bundle. Running backups with verification disabled is how a snapshot ends up streamed to the wrong endpoint.

Finally, check the server logs on the leader while the snapshot runs:

journalctl -u vault -f | grep -iE "snapshot|raft|compact"

Step-by-Step Resolution

1. Free space or move the staging path. Snapshots need headroom roughly equal to the store size. Verify before retrying:

df -h /opt/vault/data
vault operator raft snapshot save /mnt/backup/vault-$(date +%F).snap

2. Grant a dedicated snapshot policy rather than using a root token for backups:

path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}

path "sys/storage/raft/snapshot-force" {
  capabilities = ["update"]
}

Apply it and mint a scoped token for your backup job:

vault policy write raft-snapshot raft-snapshot.hcl
vault token create -policy=raft-snapshot -period=24h -orphan

3. Clean up dead servers. If autopilot state shows failed nodes, let autopilot reap them or remove the peer explicitly by its node ID:

vault operator raft autopilot state -format=json | jq '.Servers[] | {ID, Healthy, Status}'
vault operator raft remove-peer vault-node-3

Enable automatic cleanup so this does not recur:

vault operator raft autopilot set-config \
  -cleanup-dead-servers=true \
  -dead-server-last-contact-threshold=10m \
  -min-quorum=3

4. Tune log compaction if raft.db is growing out of proportion to your data. These live in the storage "raft" stanza:

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"

  # Take an FSM snapshot after this many applied logs (default 8192).
  snapshot_threshold = 8192

  # Raft log entries retained after a snapshot (default 10000).
  trailing_logs = 10000

  retry_join {
    leader_api_addr = "https://vault-node-2.internal:8200"
  }
}

Lowering trailing_logs reduces raft.db size but means a lagging follower is more likely to need a full snapshot transfer to catch up. Change one value at a time and restart nodes one at a time, standbys first.

5. Restore correctly. Point the CLI at the active node and use a token with update on the snapshot path:

export VAULT_ADDR=https://vault-node-1.internal:8200
vault operator raft snapshot restore /mnt/backup/vault-2026-07-19.snap

6. Only if the seal genuinely differs and you have accepted the consequences, use -force. Understand that this skips the check that the current seal can decrypt the snapshot:

# DANGEROUS: bypasses the seal/autoseal verification.
vault operator raft snapshot restore -force /mnt/backup/vault-2026-07-19.snap

The supported way to restore a snapshot into a cluster with a different seal is to first initialise the target cluster, then restore with the same seal configuration and unseal keys that produced the snapshot — or to perform a documented seal migration afterwards. If the snapshot came from a Shamir cluster and the target uses auto-unseal, restore into a matching Shamir cluster and migrate the seal there. A related class of failure is covered in Vault error: auto-unseal KMS access denied.

7. Verify the restore actually took. The cluster reseals during restore, so expect to unseal again on Shamir clusters:

vault status
vault operator raft list-peers
vault kv get secret/canary

(Vault Enterprise) Automated snapshots remove most of this toil. Enterprise clusters can schedule snapshots to local disk, S3, GCS, or Azure Blob directly from the server, with retention:

vault write sys/storage/raft/snapshot-auto/config/hourly \
  interval="1h" \
  retain=24 \
  storage_type="aws-s3" \
  aws_s3_bucket="vault-snapshots-prod" \
  aws_s3_region="us-east-1"

vault list sys/storage/raft/snapshot-auto/config

Prevention

• Monitor free space on the Raft data volume with an alert well before it fills — snapshots need headroom equal to the store size.
• Run backups with a scoped, periodic token carrying only sys/storage/raft/snapshot, never a root token in cron.
• Test restores on a throwaway cluster every quarter; an untested snapshot is not a backup.
• Store the unseal keys or auto-unseal key identity alongside the snapshot metadata so a restore target can be built with a matching seal.
• Turn on cleanup_dead_servers in autopilot so departed nodes cannot stall log compaction.
• Alert on vault.raft_storage.bolt.freelist and raft.db size growth so runaway storage is caught before a snapshot fails.

Related Errors

• Vault is sealed after restore — expected on Shamir clusters; the barrier reseals as part of installing a snapshot.
• failed to unseal: invalid key — the unseal keys do not match the restored barrier; see Vault error: failed to unseal.
• local node not active but active cluster node not found — no leader is elected; a quorum problem rather than a snapshot problem.
• context deadline exceeded during restore — the snapshot upload exceeded the client or proxy timeout, not a storage failure.

Frequently Asked Questions

Why does -force exist if it is so dangerous? It is meant for disaster recovery where you knowingly restore into a freshly built cluster whose seal you are about to reconcile, or for automation that has already validated key material out of band. Because it skips the seal verification, a mistake produces a cluster whose data cannot be decrypted — treat it as a last resort with a known-good copy of the snapshot held aside.

Can I take a snapshot from a standby node? No. The snapshot API is served by the active node; targeting a standby returns a redirect or local node not active. Point VAULT_ADDR at the cluster’s load balancer with request forwarding enabled, or at the current leader from vault operator raft list-peers.

Why is raft.db much larger than my actual secret data? BoltDB does not return freed pages to the filesystem, and Vault retains trailing_logs entries after each snapshot. Sustained high write volume, a lagging peer, or a dead server preventing compaction all inflate the file. Lowering trailing_logs and removing dead peers usually brings it back down after the next snapshot cycle.

Do snapshots include the audit log or the seal keys? No. A snapshot contains the encrypted storage backend and Raft configuration only. Audit device output lives wherever that device writes, and auto-unseal key material stays in your KMS. You need all three — snapshot, seal configuration, and audit retention — for a complete recovery plan. For more integrated-storage and seal fixes, see the Vault guides.