🔐 Vault Error: 'failed to decrypt encrypted stored keys' Auto-Unseal KMS Access Denied

Exact Error Message

Error initializing core: failed to check seal configuration: failed to decrypt
encrypted stored keys: error decrypting data encryption key: AccessDeniedException:
User: arn:aws:sts::111122223333:assumed-role/vault-server/i-0abc123 is not authorized
to perform: kms:Decrypt on resource: arn:aws:kms:us-east-1:111122223333:key/abcd-1234
because no identity-based policy allows the kms:Decrypt action
	status code: 400, request id: 9f2c...

On a running node you may instead see this from vault status:

Key                      Value
---                      -----
Seal Type                awskms
Initialized              true
Sealed                   true
Total Recovery Shares    5
Threshold                3
Unseal Progress          0/3
HA Enabled               true

What It Means

With auto-unseal, Vault does not hold its master key in memory across restarts. Instead the master key is encrypted with a data encryption key that is itself wrapped by a cloud KMS key (awskms, azurekeyvault, gcpckms) or an HSM (Vault Enterprise). At every startup Vault calls the KMS Decrypt operation to unwrap that key. If the call is denied, Vault has no path to its barrier and stays sealed — it does not fall back to Shamir key shares, because with auto-unseal those shares no longer exist as unseal keys.

This is an authorization failure, not a Vault configuration parse failure: the seal stanza was read correctly, credentials were found, and the API call reached the provider, which refused it. The most common shapes are a missing kms:Decrypt grant on the instance profile or workload identity, a key ARN in a different region than the one configured, or a node that was moved to a new role that never got the key policy grant. Vault will also fail on kms:Encrypt later — when it re-wraps the key on a rekey or seal migration — so grant both.

Common Causes

• The instance profile, managed identity, or GCP service account lacks kms:Decrypt (and kms:Encrypt, kms:DescribeKey) on the specific key.
• The KMS key policy does not include the Vault principal, even though the IAM identity policy does — AWS KMS requires both.
• region in the seal "awskms" stanza does not match the region the key actually lives in.
• kms_key_id points at a rotated alias, a deleted key, or a key in another account without a cross-account grant.
• Static credentials (access_key/secret_key) in the seal stanza expired or were replaced by an instance profile that has no grant.
• The key is in PendingDeletion or Disabled state, which surfaces as a denial-flavoured error rather than “not found”.

Diagnostic Commands

Confirm the node is sealed and which seal type it believes it has:

vault status
vault status -format=json | jq '{seal_type, sealed, recovery_seal, initialized}'

Read the actual startup failure — the KMS provider error is only fully rendered in the service log, not in vault status:

journalctl -u vault -n 200 --no-pager | grep -iE "seal|kms|decrypt|AccessDenied"

Identify the credential Vault is actually using. On EC2 this is the instance profile, which is frequently not the role you assumed in your shell:

TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/meta-data/iam/security-credentials/

Verify the key exists, is enabled, and is in the configured region:

aws kms describe-key --key-id alias/vault-unseal --region us-east-1 \
  --query 'KeyMetadata.{Arn:Arn,State:KeyState,Usage:KeyUsage}'

Reproduce the exact operation Vault performs, from the Vault host itself:

aws kms encrypt --key-id alias/vault-unseal --plaintext "$(echo -n probe | base64)" \
  --region us-east-1 --query CiphertextBlob --output text > /tmp/probe.b64
aws kms decrypt --ciphertext-blob fileb://<(base64 -d /tmp/probe.b64) --region us-east-1

For Azure and GCP the equivalents are:

az keyvault key show --vault-name vault-kv --name vault-unseal
gcloud kms keys describe vault-unseal --location global \
  --keyring vault --project my-project

Step-by-Step Resolution

1. Confirm the seal stanza matches reality. Region, key id, and provider must all agree with what describe-key returned:

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
}

For the other providers the equivalent stanzas are:

seal "azurekeyvault" {
  tenant_id  = "00000000-0000-0000-0000-000000000000"
  vault_name = "vault-kv"
  key_name   = "vault-unseal"
}

seal "gcpckms" {
  project    = "my-project"
  region     = "global"
  key_ring   = "vault"
  crypto_key = "vault-unseal"
}

2. Grant the identity policy the three actions Vault needs. DescribeKey is used for key validation at startup, so omitting it produces confusing partial failures:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
    }
  ]
}

3. Add the Vault principal to the key policy as well. This is the step most teams miss, because an identity policy alone is insufficient for KMS:

{
  "Sid": "AllowVaultAutoUnseal",
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::111122223333:role/vault-server" },
  "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"],
  "Resource": "*"
}

4. Remove leftover static credentials from the seal stanza if you have moved to an instance profile. Credentials in the stanza take precedence and will silently shadow the role:

grep -nE "access_key|secret_key" /etc/vault.d/vault.hcl

If you must keep them out of the config file, supply them by environment instead of inline, and make sure the unit file actually loads them:

systemctl cat vault | grep -i environment

5. Restart the service and watch the seal path specifically. A successful unwrap logs the seal type and then proceeds to unseal:

sudo systemctl restart vault
journalctl -u vault -f | grep -iE "seal|unseal|core: post-unseal"

6. Confirm the node is unsealed and the seal type is what you expect. Sealed false with a recovery-seal block is the healthy auto-unseal state:

vault status
vault operator raft list-peers

If the node still reports sealed after a clean KMS probe, check that VAULT_SEAL_TYPE is not set in the environment to something that disagrees with the config file — it overrides seal type selection during migration and is a frequent source of a node that “ignores” its stanza.

Prevention

• Manage the KMS key policy and the Vault instance role together in the same module so one can never drift from the other.
• Use a key alias rather than a raw key id in change-managed config, and pin the ARN in the seal stanza so a copied config cannot target the wrong region.
• Add a synthetic KMS encrypt/decrypt probe to node health checks so grants that are revoked are caught before the next restart.
• Store recovery keys with the same rigour as unseal keys — under auto-unseal they are what you need for rekey, root token generation, and seal migration.
• Never enable KMS key deletion or automatic rotation of the key material without first validating Vault behaviour in a non-production cluster.
• Alert on sealed=true from the /sys/health endpoint rather than relying on someone noticing failed requests.

Related Errors

• failed to unseal: invalid key — a Shamir unseal share is wrong or from another cluster, unrelated to KMS. See Vault error: failed to unseal invalid key.
• core: barrier reports initialized but no seal configuration found — the storage backend and seal stanza disagree, usually after a botched migration. See Vault error: upgrade migration failed.
• failed to create raft snapshot — storage-layer failure during snapshot, not a seal problem. See Vault error: raft snapshot failed.
• error checking seal status: connection refused — Vault is not running at all, so no seal call was ever attempted.

Frequently Asked Questions

Why can’t I just unseal with my unseal keys? With auto-unseal there are no unseal keys. Initialization produced recovery keys, which authorize privileged operations like rekey, seal migration, and root token generation — they cannot unwrap the barrier. Only the KMS can do that, which is why fixing the grant is the only path forward.

Do I need kms:Encrypt if Vault is only starting up? Startup itself needs Decrypt and DescribeKey, but Vault re-wraps the key on rekey and seal migration, and some upgrade paths re-encrypt on first boot. Granting only Decrypt produces a cluster that starts fine for months and then fails during a maintenance operation.

Is it safe to switch the seal stanza to a different key? Not by editing it in place. Changing kms_key_id without a seal migration leaves Vault unable to decrypt the existing stored keys. Use the documented seal migration flow with a quorum of recovery keys, and take a snapshot first.

Can a TLS problem masquerade as this error? Yes — if the KMS endpoint is reached through a proxy with an untrusted CA you may see a transport error nested inside the seal failure. Fix the CA trust on the host rather than disabling verification; VAULT_SKIP_VERIFY only affects Vault client connections and is a temporary diagnostic at best. More auto-unseal and seal-migration fixes are collected in the Vault guides.