☁️ OpenStack Error: 'Failed to decrypt fernet token' — fix Keystone key rotation mismatch

Exact Error Message

$ openstack server list
Failed to validate token (HTTP 401)

# keystone.log on the controller:
WARNING keystone.token.providers.fernet.token_formatters [-] This is not a
recognized Fernet payload version: invalid base64-encoded data
ERROR keystone.middleware.auth [-] Could not decrypt fernet token payload;
    cryptography.fernet.InvalidToken
WARNING keystone.common.fernet_utils [-] [fernet_tokens] key_repository does
    not appear to exist; attempting to create it

The user-facing symptom is intermittent HTTP 401 / Failed to validate token — a token works on one controller and is rejected on another.

What It Means

Keystone issues Fernet tokens that are symmetrically encrypted with keys stored in /etc/keystone/fernet-keys/. Any Keystone node that validates a token must have the same key that signed it. The key repository is an ordered set: a staged key (0), the current primary/signing key (the highest number), and several older secondary keys used only for decryption.

Could not decrypt fernet token payload means the validating node does not hold the key that encrypted the token. In a multi-controller deployment this almost always means the fernet-keys directory is out of sync between nodes, or keystone-manage fernet_rotate was run on one node without distributing the result to the others.

Common Causes

• The fernet-keys directory differs across controllers (rotation ran on only one node).
• Key rotation happened too fast — max_active_keys was exceeded and a still-valid token’s key was already deleted.
• Keys were distributed with wrong ownership/permissions and Keystone silently regenerated them.
• A node was rebuilt/added without seeding it from the existing key repository.
• Clock skew makes freshly rotated keys look expired to a lagging node.
• Someone ran fernet_setup on a running cluster, overwriting good keys with a fresh key 0.

Diagnostic Commands

Compare the key repositories across all controllers (they must be identical):

for h in ctrl-01 ctrl-02 ctrl-03; do
  echo "== $h =="; ssh $h 'ls -l /etc/keystone/fernet-keys/ && md5sum /etc/keystone/fernet-keys/*'
done

Confirm ownership and the configured repository path:

sudo crudini --get /etc/keystone/keystone.conf fernet_tokens key_repository
sudo crudini --get /etc/keystone/keystone.conf fernet_tokens max_active_keys
sudo ls -l /etc/keystone/fernet-keys/

Watch Keystone reject a token in real time:

sudo journalctl -u devstack@keystone -f   # or tail -f /var/log/keystone/keystone.log
openstack token issue

Step-by-Step Resolution

1. Pick one controller as the source of truth — the node that most recently rotated correctly and can still issue working tokens.

2. Copy its entire key repository to every other controller, preserving order and ownership:

sudo rsync -a --delete /etc/keystone/fernet-keys/ ctrl-02:/etc/keystone/fernet-keys/
sudo ssh ctrl-02 'chown -R keystone:keystone /etc/keystone/fernet-keys && chmod 600 /etc/keystone/fernet-keys/*'

3. Verify the keys are now identical everywhere:

for h in ctrl-01 ctrl-02 ctrl-03; do ssh $h 'md5sum /etc/keystone/fernet-keys/*'; done

4. Reload Keystone on each node so it re-reads the repository:

sudo systemctl reload apache2   # Keystone typically runs under Apache/mod_wsgi

5. Fix the rotation process so keys are always distributed atomically. Rotate on ONE node, then push the result — never rotate independently on each node:

sudo keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone
sudo rsync -a --delete /etc/keystone/fernet-keys/ ctrl-02:/etc/keystone/fernet-keys/

6. Confirm tokens validate everywhere:

openstack token issue -f value -c id
openstack server list   # should now return 200, not 401

Prevention

• Rotate on a single node and distribute the repository as one atomic step (Kolla, cron+rsync, or a config-management handler).
• Keep max_active_keys comfortably larger than rotation_interval / token_expiration so no valid token loses its key.
• Never run keystone-manage fernet_setup on a live cluster — it wipes existing keys. Use fernet_rotate for ongoing operation.
• Enforce identical ownership (keystone:keystone) and 0600 permissions on distributed keys.
• Run NTP/chrony on all controllers so rotation timing is consistent.

Related Errors

• Token not found (HTTP 404) — an expired or revoked token, not a decrypt failure.
• The request you have made requires authentication (HTTP 401) — missing/wrong credentials rather than a key mismatch.
• Could not find project — an authorization scope problem after auth succeeds.
• SSL exception connecting to https://...:5000 — a TLS/endpoint problem in front of Keystone.

Frequently Asked Questions

Why does the same token work on one controller and fail on another? Because that controller lacks the Fernet key that encrypted the token. The key repositories are out of sync — copy them so every node holds the same ordered keys.

How many keys should I keep? Enough that max_active_keys covers your token lifetime across the rotation interval. If keys rotate faster than tokens expire, valid tokens can outlive their signing key and start failing.

Is it safe to just delete all keys and start fresh? Running fernet_setup or clearing the directory invalidates every outstanding token, forcing all clients to re-authenticate. It fixes the mismatch but is disruptive, so distribute existing keys first when possible.

Can I automate rotation safely? Yes — rotate on one node and distribute atomically. Many teams template this with the automation patterns in the prompt library. For more identity fixes, see the OpenStack guides.