Vault Raft Cluster Sizing and Backup Prompt

You are a senior platform engineer who has run Vault on Raft integrated storage through node failures, upgrades, and one real restore, and who knows that an untested snapshot is not a backup.

I will provide:
- Where the cluster runs (VMs, Kubernetes, bare metal), across how many availability zones or datacenters
- Expected request rate, secret count, and which engines are in use (dynamic secrets and PKI drive very different write volumes)
- Availability and RPO/RTO targets
- Current node count, instance sizes, and disk type if a cluster already exists
- My Vault version and edition

Your job:

1. **Size the cluster and place the nodes.** Recommend a node count — odd numbers only, with the reasoning about quorum and fault tolerance spelled out (a 3-node cluster tolerates one failure, 5 tolerates two) — and distribute nodes across failure domains so no single zone loss costs quorum. Explicitly address the two-zone trap where any placement leaves one zone holding a majority. Give CPU, memory, and disk guidance, and stress that Raft is latency-sensitive on fsync, so low-latency local SSD matters far more than raw capacity.

2. **Write the storage and listener configuration.** Give the `storage "raft"` stanza with `path`, `node_id`, and `retry_join` blocks appropriate to my platform (address-based, or cloud auto-join by tag), plus the `listener "tcp"` block with TLS configured properly. Be explicit that `api_addr` and `cluster_addr` must be set to addresses other nodes and clients can actually reach — misconfigured advertise addresses are the single most common cause of a cluster that forms but does not work.

3. **Tune Raft parameters only where justified.** Explain `performance_multiplier`, `snapshot_threshold`, `snapshot_interval`, `trailing_logs`, and `max_entry_size`, when each should be changed, and what breaks if it's changed carelessly. Default to leaving them alone unless my workload profile gives a concrete reason.

4. **Configure autopilot.** Show `vault operator raft autopilot set-config` with `cleanup_dead_servers`, `dead_server_last_contact_threshold`, `min_quorum`, and `server_stabilization_time`, and explain how these interact with autoscaling or rolling node replacement. Make the point that `min_quorum` is the guard against autopilot cleaning its way into a quorum loss. Note that automated snapshots via `sys/storage/raft/snapshot-auto/config` are an Enterprise feature, so Community deployments must schedule snapshots externally.

5. **Design the snapshot strategy.** Give the `vault operator raft snapshot save` procedure, a schedule derived from my RPO, and where snapshots go: encrypted at rest, in a separate failure domain and separate credential boundary from Vault itself, with retention and integrity checks. Cover the identity that takes snapshots — it needs a narrowly scoped policy over the snapshot path, not root — and state clearly that a snapshot contains all Vault data in encrypted form and is only usable with the corresponding seal, which is exactly why the seal and KMS key must be protected and backed up alongside it.

6. **Write the restore procedure and require a rehearsal.** Cover both `vault operator raft snapshot restore` into an existing cluster and the full rebuild path: new cluster, matching seal configuration, restore, verify. Enumerate what a restore does not bring back — anything written after the snapshot, including dynamic credentials whose leases will be inconsistent with the real backing systems — and how to reconcile that (revoke and reissue rather than hope). Mandate a scheduled restore drill into a scratch environment with a defined success test, and say that a snapshot never restored is an assumption, not a backup.

7. **Document node lifecycle operations.** Adding a node with `retry_join`, checking membership with `vault operator raft list-peers`, removing a failed node with `vault operator raft remove-peer`, stepping down a leader with `vault operator step-down`, and the correct order for a rolling upgrade (standbys first, leader last via step-down). Include the recovery path for a cluster that has lost quorum and cannot elect a leader, and be clear that this is a dangerous procedure requiring the current state to be captured first.

8. **Specify monitoring.** Track leader elections and their frequency, per-node fsync and storage latency, Raft log and applied index lag between peers, disk usage against the Raft data path, seal state, snapshot job success, and unsealed-node count. Alert on quorum at risk — that is, healthy nodes down to threshold plus one — rather than waiting for quorum to be lost.

Output as: (a) node count, sizing, and failure-domain placement with the quorum reasoning, (b) the complete storage/listener/api_addr configuration, (c) autopilot settings with values, (d) the snapshot schedule, storage location, and scoped snapshot policy, (e) the restore procedure plus the drill plan and success test, (f) node lifecycle runbooks including quorum loss, (g) monitoring and alert thresholds.

Take a snapshot before any node-membership or upgrade operation. Do not use `remove-peer` to fix a node that is merely slow, and do not treat replication as a substitute for snapshots — replication faithfully copies a bad write to every replica.