🔐 Vault Error: 'operation not allowed on this Vault instance' on a Replication Secondary

Exact Error Message

$ vault kv put secret/app/config password=hunter2
Error making API request.

URL: PUT https://vault-dr.example.com:8200/v1/secret/data/app/config
Code: 400. Errors:

* operation not allowed on this Vault instance

On a DR secondary almost every request fails earlier, at the request-router level:

$ vault token lookup
Error making API request.

URL: GET https://vault-dr.example.com:8200/v1/auth/token/lookup-self
Code: 500. Errors:

* Vault is in a disaster recovery secondary state and cannot serve this request

What It Means

Replication is a Vault Enterprise feature, and the two modes behave very differently. A disaster recovery (DR) secondary is a warm standby: it streams all data — including tokens and leases — from the primary but serves no client requests at all. The only endpoints it answers are sys/seal-status, unseal, sys/health, and the replication/promotion endpoints under sys/replication/dr/secondary/*. Any normal read, write, or login against a DR secondary is rejected outright, which is why even a token lookup fails.

A performance secondary is different. It serves reads locally and forwards writes to the primary over the cluster port, then waits for the resulting data to stream back. It keeps its own local tokens, leases, and local auth mounts, so tokens issued on a performance secondary are not valid on the primary. You still see operation not allowed on this Vault instance when you write to a path that cannot be forwarded — for example a path excluded by a paths filter, or a write attempted while the connection to the primary is down so there is nothing to forward to. Both failure classes come down to the same question: which cluster am I actually talking to, and is its replication link healthy?

Common Causes

• A client, load balancer, or DNS record is pointing at a DR secondary instead of the primary — DR secondaries serve nothing.
• A write was issued to a performance secondary while connection_state was not ready, so it could not be forwarded to the primary.
• The mount or namespace is excluded by a paths filter (formerly called mount filters) on that performance secondary, so the data does not exist there.
• The cluster port 8201 is blocked between clusters, or cluster_addr/api_addr advertise an address the peer cannot reach.
• The secondary was activated with a stale or already-consumed activation token and never completed its initial sync.
• The secondary is still streaming its initial merkle sync, so it reports merkle-diff or merkle-sync rather than stream-wals.

Diagnostic Commands

First, establish what the node actually is. Do this on both clusters — the answer is frequently a surprise:

vault read -format=json sys/replication/status | jq '{dr: .data.dr.mode, perf: .data.performance.mode}'

Then pull the full status on the secondary. This is the single most useful command for replication problems:

vault read -format=json sys/replication/status | jq '.data.dr'

A healthy DR secondary looks roughly like this:

{
  "mode": "secondary",
  "state": "stream-wals",
  "connection_state": "ready",
  "primary_cluster_addr": "https://vault-primary.example.com:8201",
  "known_primary_cluster_addrs": ["https://vault-primary.example.com:8201"],
  "last_remote_wal": 148213,
  "merkle_root": "3f9a...",
  "secondary_id": "dr-eu-west-1"
}

The state field is the one to read carefully. stream-wals means steady-state replication. merkle-diff means the two clusters are comparing merkle trees to find divergence, and merkle-sync means it is actively re-syncing the differing subtrees — normal after activation or a long outage, but a cluster stuck there for hours indicates a throughput or connectivity problem. idle means it is not replicating at all.

Compare merkle roots across clusters. Identical roots mean the data sets agree:

# On the primary
vault read -field=merkle_root sys/replication/status
# On the secondary
vault read -field=merkle_root sys/replication/status

Verify the cluster port is actually reachable from the secondary. Replication uses the cluster address (default port 8201), not the API port 8200:

nc -vz vault-primary.example.com 8201
openssl s_client -connect vault-primary.example.com:8201 -alpn req_fw_sb-act_v1 </dev/null 2>&1 | head -20

If the TLS handshake fails here but the API port works, you have a certificate or SAN problem on the cluster listener rather than a replication bug. You can temporarily add -tls-skip-verify to a vault status call purely to confirm the hypothesis that trust is the issue — never leave it in place, and restore proper CA trust as the actual fix. See Vault error: failed to unseal — invalid key for the related seal-side checks.

Finally, list what is actually replicated to a performance secondary:

vault read -format=json sys/replication/performance/primary/paths-filter/perf-eu 2>/dev/null | jq '.data'
vault secrets list -detailed

Step-by-Step Resolution

1. Confirm you are talking to the intended cluster before changing anything. A DR secondary is supposed to reject your write:

echo "$VAULT_ADDR"
vault read -field=mode sys/replication/status

If the answer is secondary under dr, repoint VAULT_ADDR at the primary and stop — there is no bug to fix.

2. If the primary is genuinely lost and you need to promote the DR secondary, generate a DR operation token. Promotion requires it; a normal root token will not do. It uses the same generate-root style workflow, driven by unseal/recovery key holders:

vault operator generate-root -dr-token -init
# distribute the returned nonce + one-time password (OTP) to key holders
vault operator generate-root -dr-token -nonce=<nonce>   # each key holder pastes a key share
vault operator generate-root -dr-token -otp=<otp> -decode=<encoded_token>

3. Promote the DR secondary with that token. After promotion the cluster becomes a full primary and starts serving requests:

vault write -f sys/replication/dr/secondary/promote dr_operation_token="$DR_OP_TOKEN"
vault read -field=mode sys/replication/status   # expect: primary

4. If a secondary never completed activation, re-enable it with a fresh activation token. Activation tokens are single-use and short-lived, so always mint a new one rather than reusing an old value:

# On the primary
vault write sys/replication/dr/primary/secondary-token id=dr-eu-west-1

# On the secondary (this wipes the secondary's data and re-syncs from the primary)
vault write sys/replication/dr/secondary/enable token="$ACTIVATION_TOKEN" \
  primary_api_addr="https://vault-primary.example.com:8200"

5. Fix the cluster address advertisement if connection_state is not ready. The secondary connects to whatever cluster_addr the primary advertises, so a value like 127.0.0.1:8201 behind a load balancer will never work:

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/etc/vault/tls/vault.crt"
  tls_key_file    = "/etc/vault/tls/vault.key"
}

api_addr     = "https://vault-primary.example.com:8200"
cluster_addr = "https://vault-primary.example.com:8201"

Restart the node after changing these, and confirm the peer now lists the correct value in known_primary_cluster_addrs.

6. If a write fails on a performance secondary because the mount is filtered out, adjust the paths filter on the primary rather than writing locally. Paths filters are defined per-secondary and take a mode of allow or deny:

vault write sys/replication/performance/primary/paths-filter/perf-eu \
  mode=allow paths="secret/,eu-only/"

Then re-check sys/replication/status on the secondary until state returns to stream-wals.

Prevention

• Never point application load balancers or service discovery at DR secondaries; use a separate hostname that only operators know.
• Alert on sys/replication/status when connection_state leaves ready or state leaves stream-wals for more than a few minutes.
• Open and monitor port 8201 between clusters explicitly, and include the cluster hostnames in the certificate SANs.
• Rehearse DR promotion on a schedule, including the DR operation token generation, so key holders know the workflow under pressure.
• Set api_addr and cluster_addr explicitly in every node’s config rather than relying on auto-detection behind NAT or load balancers.
• Document which mounts are covered by paths filters so nobody debugs a “missing secret” that was intentionally excluded.

Related Errors

• Vault is sealed — the secondary is up but unsealed keys were never applied; a seal problem, not replication. See Vault error: auto-unseal KMS access denied.
• namespace not found — the namespace exists on the primary but is filtered out or not yet synced to this secondary. See Vault error: namespace not found.
• local node not active but active cluster node not found — intra-cluster forwarding failure, distinct from cross-cluster replication.
• failed to take snapshot — Raft snapshot machinery, a separate storage-layer concern. See Vault error: Raft snapshot failed.

Frequently Asked Questions

Can I read secrets from a DR secondary? No. A DR secondary answers only unseal, health/seal-status, and its own replication and promotion endpoints. If you need read-serving standbys in another region, that is performance replication, not DR — and both are Vault Enterprise features.

Why do my tokens stop working after promoting a performance secondary? Performance secondaries maintain their own local tokens and leases, which are not replicated back to the primary. DR secondaries, by contrast, replicate tokens and leases, so a promoted DR secondary keeps existing sessions valid.

How long should merkle-sync take? It scales with the size of the divergent subtree and the bandwidth between clusters, so minutes for a small delta and potentially hours after a long partition. If last_remote_wal is not advancing at all, the problem is connectivity on port 8201, not sync volume.

Can I reuse a secondary activation token? No — they are single-use and short-TTL by design. Generate a fresh one with sys/replication/dr/primary/secondary-token each time you enable or re-enable a secondary. For more replication and Enterprise fixes, see the Vault guides.