🔐 Vault Error: 'failed to audit request' Audit Device Blocking Every Request

Exact Error Message

$ vault kv get secret/app/config
Error making API request.

URL: GET https://vault.internal:8200/v1/secret/data/app/config
Code: 500. Errors:

* failed to audit request, cannot continue

The server log carries the underlying cause:

[ERROR] audit: backend failed to log request: backend=file/ error="write /var/log/vault/audit.log: no space left on device"
[ERROR] core: failed to audit request: path=secret/data/app/config error="no audit backend succeeded in logging the request"

You may instead see the response side fail, after the operation already executed:

Code: 500. Errors:

* failed to audit response, cannot continue

Or, with a socket audit device whose collector has gone away:

[ERROR] audit: backend failed to log response: backend=socket/ error="dial tcp 10.4.2.19:9090: connect: connection refused"

What It Means

Vault treats auditability as a hard precondition, not a best-effort side channel. Before it processes a request, it writes an audit entry to every enabled audit device. If at least one device succeeds, the request proceeds. If every enabled device fails, Vault refuses the request and returns 500 failed to audit request, cannot continue. This is deliberate: a secrets broker that silently serves credentials it cannot record is worse than one that is briefly unavailable. The consequence is that a single misconfigured audit device — when it is your only one — converts a full disk into a complete cluster outage, with every application that reads secrets failing simultaneously.

The same rule applies to responses. A response-side failure is more painful because the operation has already been applied to storage; the client gets a 500 and does not learn what happened, but the write took effect. Note also that Vault’s file audit device holds an open file handle. If log rotation moves or deletes the file without telling Vault, the daemon keeps writing to an unlinked inode — disk fills with a file nobody can see, and ls shows a fresh empty audit.log. Vault reopens its file audit device handles on SIGHUP, which is what makes postrotate hooks essential.

Common Causes

• The filesystem holding the audit log is full — audit logs are verbose and grow fast under load.
• Log rotation deleted or moved the file without sending SIGHUP, so Vault holds a stale file handle.
• A socket audit device points at a collector (Fluentd, Logstash, Vector) that has crashed or been redeployed.
• A syslog audit device fails because the local syslog daemon is down, or its socket buffer is full.
• Only one audit device is enabled, removing all redundancy — the core anti-pattern behind this outage class.
• Filesystem permissions changed so the Vault service user can no longer write to the audit path.

Diagnostic Commands

First, find out what is enabled. The -detailed flag shows each device’s options, which is where the failing path or address lives:

vault audit list -detailed

Note that this command itself is an API request and is therefore subject to the same audit gate. If audit is fully broken, it will also return 500. When that happens, read the server logs directly instead:

journalctl -u vault --since "15 min ago" | grep -iE "audit|no space|connection refused"

Check disk on every path an audit device writes to:

df -h /var/log/vault
du -sh /var/log/vault/*

Look for the classic deleted-but-open file, which shows disk usage that du cannot explain:

lsof -p "$(pgrep -x vault)" | grep -i deleted

If a socket device is configured, test reachability to the collector from the Vault host:

nc -vz 10.4.2.19 9090
ss -tanp | grep 9090

And for a syslog device, confirm the local daemon is actually accepting messages:

systemctl status rsyslog
logger -t vault-probe "audit path probe"
journalctl -t vault-probe --since "1 min ago"

Confirm the Vault service user can still write where it thinks it can:

ls -ld /var/log/vault
sudo -u vault touch /var/log/vault/.writeprobe && sudo -u vault rm /var/log/vault/.writeprobe

Step-by-Step Resolution

1. Restore write capability first — this is the fastest path back to service. Free space on the audit volume:

df -h /var/log/vault
sudo find /var/log/vault -name 'audit.log.*.gz' -mtime +7 -delete
df -h /var/log/vault

2. If a rotation left a stale handle, send SIGHUP to make Vault reopen its file audit devices. This does not restart Vault, does not reseal it, and does not drop connections:

sudo systemctl reload vault
# or, equivalently:
sudo kill -HUP "$(pgrep -x vault)"

3. If the device cannot be repaired quickly — a dead socket collector, an unrecoverable volume — disable the broken device to unblock traffic. This requires a root or a suitably privileged operator token, and this is the moment you discover whether you have one. Keep a break-glass token or the unseal/recovery keys stored out-of-band precisely for this:

vault audit list -detailed
vault audit disable socket/

Because sys/audit operations are themselves audited, you may need to disable the failing device while another device is still working — which is the whole argument for running more than one. If every device is failing, the sys/audit write can also be blocked; in that case repair the fastest-to-fix device (usually a file device on a freshly cleared path) rather than trying to disable your way out.

4. Add a second, independent audit device so a single failure can never block requests again. Put them on different failure domains — different volume, different transport:

vault audit enable -path=file-primary file \
  file_path=/var/log/vault/audit.log

vault audit enable -path=file-standby file \
  file_path=/var/audit-standby/vault-audit.log

vault audit enable -path=syslog-fallback syslog \
  tag="vault" facility="AUTH"

A socket device should always carry a non-blocking failure mode when it is not your only device:

vault audit enable -path=socket socket \
  address=10.4.2.19:9090 \
  socket_type=tcp

5. Fix rotation so this cannot recur. The postrotate hook must signal Vault; copytruncate is the alternative if you cannot signal, but it risks losing entries written between the copy and the truncate:

/var/log/vault/audit.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    create 0600 vault vault
    postrotate
        /bin/systemctl reload vault > /dev/null 2>&1 || true
    endscript
}

6. Verify the recovery end to end, and confirm entries are landing in both devices:

vault audit list -detailed
vault kv get secret/app/config
sudo tail -n 2 /var/log/vault/audit.log | jq '{type, "path": .request.path, "time": .time}'
sudo tail -n 2 /var/audit-standby/vault-audit.log | jq '.type'

A sample entry shows the HMAC treatment that makes these logs safe to ship:

{
  "time": "2026-07-19T14:22:05.113Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:9f2c...",
    "accessor": "hmac-sha256:41ab...",
    "display_name": "approle",
    "policies": ["default", "app-reader"]
  },
  "request": {
    "operation": "read",
    "path": "secret/data/app/config",
    "remote_address": "10.4.1.77"
  }
}

Vault HMACs sensitive fields — tokens, and by default the accessor — with a per-cluster salt, so the log is correlatable without being a credential dump. If you need to match audit entries against vault token lookup-accessor output, set hmac_accessor=false on the device so accessors are written in the clear. Do that only where the log’s own access controls are strong, and never as a blanket default.

Prevention

• Always enable at least two audit devices on independent failure domains; one device is a single point of total outage.
• Alert on free space for every audit path at 20% remaining, well before Vault starts failing requests.
• Put audit logs on their own volume so a noisy neighbour cannot fill the disk Vault depends on.
• Signal Vault with SIGHUP from every rotation hook, and test rotation in staging rather than discovering it at 3am.
• Keep a break-glass root or operator token sealed out-of-band, since fixing audit failures requires privileged access you cannot mint while audit is down.
• Health-check socket audit collectors and page on them; a redeployed log pipeline should never be able to take Vault offline.

Related Errors

• local node not active but active cluster node not found — a leadership problem, not an audit gate, though both present as cluster-wide 500s.
• Vault is sealed — the barrier is closed; audit devices are not consulted until Vault is unsealed. See Vault error: failed to unseal.
• rate limit quota exceeded — request rejection by quota rather than by audit failure; see Vault error: rate limit quota exceeded.
• permission denied on sys/audit/ — your operator token lacks the capability needed to disable the failing device.

Frequently Asked Questions

Why does Vault refuse requests instead of just dropping the audit entry? Because an unlogged secret access is unauditable forever. Vault’s design position is that availability must not silently trade away the audit trail, so it fails closed. The correct mitigation is redundancy — with two healthy devices, one failure is invisible to clients.

Can I disable a broken audit device without a root token? You need a token with update/sudo on sys/audit/<path>. That is deliberately privileged. Pre-create an operator policy granting exactly that and store an associated token out-of-band, because you cannot log in to mint a new one while every audit device is failing.

Does SIGHUP restart Vault or reseal it? No. It reloads certain configuration and reopens file audit device handles. Active connections stay up and the barrier stays unsealed, which is what makes it safe to run from a logrotate postrotate hook.

Should I turn off HMAC for accessors or other fields? Only with a clear reason. Setting hmac_accessor=false makes accessors greppable for incident correlation, and log_raw=true disables hashing entirely — the latter writes sensitive request data in plaintext and is almost never appropriate outside a short, contained debugging window. For more Vault operational fixes, see the Vault guides.