Vault Error: 'permission denied' Reading or Writing a Path
Fix Vault's 403 'permission denied': audit ACL policy capabilities, handle KV v2 data/ and metadata/ prefixes, use token capabilities, and resolve deny rules and sudo paths.
- #vault
- #secrets
- #security-hardening
- #troubleshooting
- #errors
Stuck on this HashiCorp Vault error? Get the free incident triage checklist
A one-page PDF — the exact steps to isolate, fix, and verify a production error like this one. No spam, unsubscribe anytime.
Exact Error Message
$ vault kv get secret/app/config
Error making API request.
URL: GET https://vault.example.com/v1/secret/data/app/config
Code: 403. Errors:
* 1 error occurred:
* permission denied
A write to a path the token can only read fails the same way:
$ vault write secret/data/app/config data=@payload.json
Error writing data to secret/data/app/config: Error making API request.
URL: PUT https://vault.example.com/v1/secret/data/app/config
Code: 403. Errors:
* 1 error occurred:
* permission denied
What It Means
Vault authenticated your token successfully and then refused the operation. Every request is evaluated against the union of the ACL policies attached to the token: Vault takes the request path, matches it against the policy rules, and checks whether the HTTP verb’s corresponding capability is granted. A GET needs read, a PUT/POST needs create or update, a LIST needs list, and a DELETE needs delete. If no rule matches the path, or a matching rule lacks the capability, you get 403 permission denied.
This is distinct from an authentication failure. A missing or malformed token produces a 400 with missing client token, and an expired or revoked token produces a 403 with permission denied too — but vault token lookup will fail outright in that case. If lookup succeeds and the path still 403s, the problem is the policy, not the credential. The single most common cause on KV version 2 mounts is a policy written against the human-facing path (secret/app/config) when the API path Vault actually evaluates is secret/data/app/config.
Common Causes
- The policy grants
readbut the operation is a write, or grantslistbut notread(or vice versa). - A KV v2 policy omits the
data/segment, so the rule never matches the real API path. - Listing KV v2 keys is denied because
listwas granted ondata/instead ofmetadata/. - An explicit
denyin one attached policy overrides anallowin another — deny always wins. - The path is root-protected and the rule lacks the
sudocapability. - A glob or
+wildcard does not match the way you assumed, so the intended rule is never applied.
Diagnostic Commands
Ask Vault directly what the token can do on the exact path. This is the fastest route to an answer:
vault token capabilities secret/data/app/config
Inspect the token to see which policies are actually attached and whether it is still valid:
vault token lookup -format=json | jq '.data | {policies, identity_policies, ttl, path}'
Read the policy body so you can compare its rules against the failing path:
vault policy list
vault policy read app-readonly
Confirm the mount type and version, since KV v1 and KV v2 use different API paths:
vault secrets list -detailed | grep -E "^Path|secret/"
Check the audit log for the rejected request, which records the exact path and operation Vault evaluated:
sudo grep '"error":"permission denied"' /var/log/vault/audit.log | tail -5 | jq '.request | {path, operation}'
Step-by-Step Resolution
- Determine the true API path. The CLI hides the KV v2 rewrite, so run the command with
-output-curl-stringto see what is really sent:
vault kv get -output-curl-string secret/app/config
# curl -H "X-Vault-Token: $(vault print token)" \
# https://vault.example.com/v1/secret/data/app/config
- Run a capabilities check on that exact path. If the result is
deny, the policy does not cover it:
vault token capabilities secret/data/app/config
# deny
- Write a policy against the API path, including both
data/for values andmetadata/for listing and versions:
# app-readonly.hcl
path "secret/data/app/*" {
capabilities = ["read"]
}
path "secret/metadata/app/*" {
capabilities = ["list", "read"]
}
- Grant the capability that matches the operation. The full set is
create,read,update,patch,delete,list,sudo, anddeny. Usepatchforvault kv patchon KV v2, and remembercreateandupdateare separate — a token with onlycreatecannot overwrite an existing key:
path "secret/data/app/*" {
capabilities = ["create", "read", "update", "patch", "delete"]
}
path "secret/delete/app/*" {
capabilities = ["update"]
}
path "secret/destroy/app/*" {
capabilities = ["update"]
}
- Apply the policy and attach it to the auth method role or the token, then re-authenticate so a fresh token picks it up. Editing a policy does not require reissuing tokens — policies are evaluated per request — but adding a new policy to a role does require a new login:
vault policy write app-readonly app-readonly.hcl
vault write auth/approle/role/app-role token_policies="app-readonly"
vault login -method=approle role_id="$ROLE_ID" secret_id="$SECRET_ID"
vault token capabilities secret/data/app/config
# read
- If capabilities still show
deny, look for an explicitdenyrule or a wildcard mismatch. Adenyanywhere in any attached policy beats every allow, and the more specific path prefix wins over a glob. Note that*is only valid as the final character of a path and matches everything after it, while+matches exactly one path segment:
# matches secret/data/app/prod/db but not secret/data/app/prod/db/replica
path "secret/data/app/+/db" {
capabilities = ["read"]
}
# matches everything under app/, at any depth
path "secret/data/app/*" {
capabilities = ["read"]
}
# this wins over both of the above for its path
path "secret/data/app/prod/root" {
capabilities = ["deny"]
}
For dynamic per-user paths, use a templated policy so one policy serves every entity rather than generating one policy per user:
path "secret/data/users/{{identity.entity.id}}/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
Root-protected endpoints — sys/audit, sys/rotate, sys/seal, and parts of auth/token — additionally require sudo alongside the normal capability. If the token was fine yesterday and lookup now fails as well, the credential itself has lapsed; see Vault error: “token expired / invalid token”. And if the path genuinely has nothing stored at it, Vault answers 404 rather than 403, covered in Vault error: “No value found at path”.
Prevention
- Always write KV v2 policies against
data/,metadata/,delete/,destroy/, andundelete/prefixes explicitly. - Keep policies in version control and apply them through CI with
vault policy writerather than by hand. - Add a
vault token capabilitiesassertion to your deployment smoke tests for each critical path. - Reserve
denyfor narrow carve-outs and document why, since it silently overrides every other policy. - Grant
sudoonly on the specific root-protected paths that need it, never on a broad glob. - Prefer templated policies with
identity.entity.idover per-user policies that drift out of sync.
Related Errors
missing client token— no credential was presented at all; a 400, not an authorization failure.no handler for route— the mount does not exist or the path is misspelled, so no policy is even consulted.1 error occurred: * permission deniedonsys/paths — usually a missingsudocapability rather than a missing verb.preflight capability check returned 403— the KV helper checkedmetadata/and was denied before reachingdata/.
Frequently Asked Questions
Why does vault kv get secret/app/config fail when my policy says path "secret/app/*"? On a KV v2 mount the real API path includes a data/ segment, so the rule must be secret/data/app/*. The CLI’s friendly path is not what the policy engine sees.
I granted list but vault kv list still fails. On KV v2, listing reads the metadata tree. Grant list on secret/metadata/app/*, not on secret/data/app/*.
Can one policy’s allow override another’s deny? No. Deny always wins across the union of attached policies, including identity-group policies. Find the offending rule with vault policy read on every policy in vault token lookup.
What’s the difference between * and + in a policy path? + matches a single path segment and can appear anywhere; * is a suffix wildcard that matches the remainder of the path and is only valid as the last character. More policy and auth walkthroughs live in the Vault guides.
Fixed it? Get 500 HashiCorp Vault & DevOps AI prompts — free
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.
Did this fix your issue?
Get 500 Battle-Tested DevOps AI Prompts — Free
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.