☁️ OpenStack Error: Keystone 'Failed to validate token' Token Not Found (404)

Exact Error Message

$ openstack server list
The request you have made requires authentication. (HTTP 401)

# with --debug, the underlying validation failure is a 404 from Keystone:

Failed to validate token (HTTP 404) (Request-ID: req-a1b2c3d4-...)
keystoneauth1.exceptions.http.NotFound: Could not find token: gAAAAABm...  (HTTP 404)

In the keystone server log the corresponding line is:

WARNING keystone.common.wsgi [req-...] Could not find token: gAAAAABm7Qk...:
    keystone.exception.TokenNotFound: Could not find token: gAAAAABm7Qk...

What It Means

Every OpenStack API call carries an authentication token in the X-Auth-Token header. When a service (Nova, Neutron, Cinder) receives a request, it asks Keystone to validate that token. A 404 / TokenNotFound means Keystone was reached but does not recognize the token as valid — it has expired, was revoked, or cannot be decrypted.

With the default fernet token provider, tokens are not stored in a database; they are encrypted payloads. So “not found” usually means Keystone could not decrypt or verify the token — most often because of expiry, a fernet key mismatch across nodes, or clock skew — rather than a missing database row.

Common Causes

• The token has simply expired (default lifetime is one hour) and the client cached it.
• Fernet key repositories differ between Keystone nodes, so a token minted on node A cannot be validated on node B behind the load balancer.
• Clock skew between Keystone nodes makes a freshly issued token look already expired.
• The token was explicitly revoked (password change, project role change, openstack token revoke).
• Stale OS_AUTH_URL or a cached catalog pointing at an old/wrong Keystone endpoint.
• A restored database or rotated keys invalidated all previously issued tokens.

Diagnostic Commands

Reproduce with debug to see the real 404 behind the 401:

openstack --debug token issue 2>&1 | grep -iE "404|token|auth_url"

Confirm you can get a fresh token at all:

env | grep OS_
openstack token issue -f value -c id

Check fernet key consistency across every Keystone node (the SHA sums must match):

sudo ls -l /etc/keystone/fernet-keys/
sudo sha256sum /etc/keystone/fernet-keys/* 

Check for clock skew on the Keystone hosts:

timedatectl status | grep -E "synchronized|NTP"

Inspect the catalog to be sure endpoints are current:

openstack catalog list
openstack endpoint list --service identity

Step-by-Step Resolution

1. Get a brand-new token and retry. If a fresh token works and the old one did not, you were simply holding an expired/revoked token — clear any cached one:

unset OS_TOKEN
openstack token issue -f value -c id
openstack server list

2. If a fresh token still 404s intermittently behind a load balancer, suspect fernet key drift. Compare the key repositories on all Keystone nodes:

sudo sha256sum /etc/keystone/fernet-keys/0 /etc/keystone/fernet-keys/1

They must be identical on every node. If they differ, resynchronize the repository (rsync from the primary or re-run your key-distribution automation), then reload Keystone:

sudo systemctl reload apache2   # or httpd / wsgi container

3. Fix clock skew, which is a frequent cause of “expired the instant it was issued.” Ensure NTP is synchronized on all Keystone nodes:

sudo systemctl enable --now chrony
chronyc sources

4. If keys were rotated correctly but you rotated too aggressively, remember fernet needs enough staged keys to cover the token lifetime. Verify the rotation count and interval:

grep -E "max_active_keys|expiration" /etc/keystone/keystone.conf

max_active_keys should be at least (token_expiration / rotation_interval) + 2.

5. If the endpoint is wrong or stale, point the client at the correct identity URL and re-authenticate:

export OS_AUTH_URL=https://keystone.example.com:5000/v3
openstack token issue

6. Confirm end-to-end by calling a downstream service that must validate the token with Keystone:

openstack server list
openstack volume list

Prevention

• Distribute fernet keys atomically to every Keystone node (rotate on one, sync to all) and never let repositories drift.
• Run NTP on all identity nodes and alert on skew; token validity is time-sensitive.
• Size max_active_keys to cover the full token lifetime plus a rotation buffer before automating rotation.
• Do not cache tokens longer than their lifetime in scripts; re-issue or use application credentials.
• Health-check each Keystone node behind the load balancer so a node with bad keys is pulled out of rotation.

Related Errors

• The request you have made requires authentication (HTTP 401) — the client-facing wrapper around a 404 token validation.
• Failed to decrypt fernet token — a more specific key-mismatch failure during validation.
• You are not authorized to perform the requested action (HTTP 403) — authentication succeeded but authorization (policy/RBAC) failed.
• Unable to establish connection to keystone — a reachability problem rather than an invalid token.

Frequently Asked Questions

Why a 404 and not a 401 from Keystone? Keystone returns 404 TokenNotFound because, to the validation endpoint, the token simply is not a known/valid token. Downstream clients then surface it to you as a 401.

I only see this behind my load balancer — why? That is the classic fernet key-drift symptom: a token minted by one node cannot be decrypted by another node with a different key repository. Sync the keys across all nodes.

Do fernet tokens live in the database? No. They are self-contained encrypted payloads, so “not found” almost always means expiry, revocation, or a key/clock problem rather than a missing row.

Can I get a guided diagnosis from the debug output? Yes — drop the --debug trace into the DevOps AI prompt library to pinpoint whether it is expiry, key drift, or skew.

Where can I find more Keystone fixes? See the full OpenStack guides for related identity and RBAC troubleshooting.