🔑 Migrating Keystone From Fernet to JWS Tokens in OpenStack

Fernet tokens solved Keystone’s old “tokens in a database” scaling problem, but they came with a tax I’d grown tired of paying: a symmetric key repository that every Keystone node must hold identically, rotated in lockstep, distributed by rsync or config management, and capable of breaking every login if two nodes drift. JWS tokens — JSON Web Signature, asymmetric — trade that shared-secret distribution for a signing keypair, which changes the operational shape of key management. Migrating between them looks like a one-line config change and absolutely is not. Here’s the runbook I use, and how AI helps me validate each step without ever holding a credential.

What Actually Changes Between Fernet and JWS

Both Fernet and JWS produce non-persistent tokens — Keystone doesn’t store them in a database, it validates them cryptographically. The difference is the key model. Fernet uses a symmetric key repository: every node encrypts and decrypts with the same set of keys, so all nodes must have identical key material. JWS uses an asymmetric keypair: Keystone signs with a private key and anyone can validate with the public key, which changes how you distribute and rotate material.

Look at your current setup first:

grep -E '^\[token\]|provider' /etc/keystone/keystone.conf
ls -la /etc/keystone/fernet-keys/

The provider line is what you’ll flip. But the real work is in the surrounding choreography, because all Keystone nodes behind your load balancer must agree on the provider and have valid keys at the same time. The openstack category collects related identity playbooks.

Generating and Distributing the JWS Keypair

JWS uses a key repository too, but it holds an asymmetric keypair rather than symmetric keys. Keystone provides the setup helper:

keystone-manage create_jws_keypair --keystone-user keystone --keystone-group keystone
ls -la /etc/keystone/jws-keys/

You then distribute that same keypair to every Keystone node — the private key signs, the public key validates, and every node needs both so any node can issue and any node can verify. This is where your existing Fernet distribution mechanism (rsync, Barbican, Ansible) gets repurposed. Map the new flow against the old one explicitly so you don’t leave a node behind.

Prompt: “Here’s how I currently distribute Fernet keys (Ansible pushing to /etc/keystone/fernet-keys on 3 nodes) and my LB config fronting those 3 Keystone nodes. Design the equivalent JWS keypair distribution, list which nodes must have the keypair before I flip the provider, and produce a node-by-node cutover order that keeps the LB pool serving valid tokens at every step. Mark any step where two nodes would disagree on provider as a hard stop. Do not include my key material in your reasoning.”

Output: A six-step runbook that distributed the JWS keypair to all three nodes first, then drained-flipped-validated-readded one node at a time, with an explicit “do not flip a node still in the LB pool” gate and a note that long-lived Heat trusts would survive because JWS tokens validate the same trust records.

That cutover ordering is the part AI is genuinely good at: it’s pure sequencing against a stated constraint, and the model held the LB-consistency rule the whole way through. I still verify the trust-survival claim myself, because token-reuse consumers are exactly where a confident-but-wrong answer hurts.

The Cutover: Drain, Flip, Validate, Re-add

The non-negotiable rule: never flip the token provider on a node that’s still in the load-balancer pool serving live traffic. If node A is on JWS and node B is on Fernet, a client whose request lands on B can’t validate a token issued by A, and you get intermittent 401s across every service in the cloud — the worst kind of bug, because it’s not constant.

So, per node:

# 1. drain from LB (mark node down in haproxy/your LB)
# 2. flip the provider
sed -i 's/provider = fernet/provider = jws/' /etc/keystone/keystone.conf
systemctl restart httpd   # or apache2 / uwsgi

# 3. validate against the drained node directly
openstack --os-auth-url https://<this-node>:5000/v3 token issue

# 4. re-add to LB only after validation passes

Pro Tip: Validate by pointing --os-auth-url directly at the drained node, bypassing the LB. That proves the node itself works before it serves real users. Validating through the LB tells you nothing about which node answered.

Handling In-Flight Tokens and Trusts

When you flip a node, tokens already issued under Fernet won’t validate on a JWS node. You have two strategies: wait out the token TTL so old tokens expire naturally before you complete the migration, or accept a brief window where some users re-authenticate. The TTL-drain approach is gentler, but it means the migration spans at least one token lifetime.

The sharp edge is long-lived consumers: Heat trusts, autoscaling, and any service caching a token for reuse. Inventory them before you start:

openstack trust list --os-cloud admin

When I’m reasoning about which consumers might break, I’ll hand the trust list and service inventory to Claude and ask it to flag anything that reuses a token across the cutover window. That triage is helpful, but I verify each flagged consumer by hand, because “this autoscaling group reuses a trust” is a claim I need to be certain about, not 80% confident. Reusable migration prompts live in the prompt workspace.

Keeping Rollback Trivial

Because both providers are stateless and key-based, rollback is a config flip if you keep the Fernet key repository intact. Do not delete /etc/keystone/fernet-keys/ until JWS is validated across all nodes and has run clean for a while. With the old keys in place, reverting is the same drain-flip-validate-readd dance in reverse — not a key-material restore from backup, which would be far scarier.

Conclusion

Migrating Keystone from Fernet to JWS is a sequencing problem wearing a config-change costume. The risk isn’t the provider line; it’s the moment two load-balanced nodes disagree, or a long-lived trust breaks, or you delete the Fernet keys before you’re sure. AI shines at the sequencing and the dependency triage: cutover ordering, trust inventory, LB-consistency checks. None of those replace pointing openstack token issue at each drained node yourself and keeping the old keys until you’re done. The model drafts the choreography; you execute it and keep the credentials on your side of the line. For more Keystone prompts, see the prompts library.