🔐 Securing Secrets with Barbican Key Management in OpenStack

Barbican is the OpenStack service almost nobody deploys until they need it — and then several other services need it at once. It’s the key manager: it stores TLS certificates for Octavia, encryption keys for Cinder volume encryption and Nova ephemeral encryption, and any other secret a tenant or service wants managed centrally. After years of running it, I’ll tell you the hard truth up front: the most dangerous thing about Barbican isn’t a bug, it’s a misconfigured kek (key encryption key) that quietly makes every stored secret unrecoverable. Get the crypto backend right first; everything else is ordinary debugging.

What Barbican actually stores, and how

Barbican stores secrets encrypted, and the encryption is done by a pluggable secret store backend. The choices matter enormously:

• simple_crypto — Barbican encrypts secrets with a single KEK defined in barbican.conf. Easy to deploy, but that KEK is your master key. Lose it and every secret is gone; leak it and every secret is exposed.
• PKCS#11 / HSM — secrets are wrapped by keys inside a hardware security module (or SoftHSM for testing). This is what you want in production.
• KMIP — talks to an external enterprise key manager.

Know which one you’re running before you touch anything:

grep -E 'enabled_secretstore_plugins|encryption' /etc/barbican/barbican.conf

Step 1: Confirm Barbican is healthy end to end

Do a round-trip test — store and retrieve a secret — before debugging anything downstream:

openstack secret store --name test-secret \
  --payload "hello" --secret-type passphrase
openstack secret list
openstack secret get <secret-href> --payload

If the store succeeds but the get fails to return the payload, your crypto backend can’t decrypt — which usually means the KEK or HSM key changed or is unreachable. That’s the failure to fear, because it means existing secrets are at risk too. If both succeed, Barbican core is fine and the problem is in whoever’s calling it.

Step 2: The simple_crypto KEK trap

With simple_crypto, the KEK is a base64 value in the config:

[simple_crypto_plugin]
kek = 'YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXowMTIzNDU='

Three rules I enforce: the KEK must be identical on every Barbican node (a per-node KEK means secrets stored on node A can’t be read on node B — random decrypt failures under load), it must be backed up somewhere offline and access-controlled, and you must never rotate it casually, because rotating the KEK without re-encrypting existing secrets bricks them. If you’re running simple_crypto in production at all, plan the migration to PKCS#11 — but until then, treat that base64 string like the root password it effectively is.

Step 3: Debugging the HSM/PKCS#11 backend

With the HSM plugin, decrypt failures usually mean the module can’t reach the HSM or the slot/token credentials are wrong:

[p11_crypto_plugin]
library_path = /usr/lib/libCryptoki2_64.so
token_label = barbican
slot_id = 1

Check the Barbican API log for PKCS#11 errors:

journalctl -u barbican-api -f
# or in Kolla:
docker logs barbican_api 2>&1 | tail -100

CKR_TOKEN_NOT_PRESENT or CKR_PIN_INCORRECT are HSM-side: wrong slot, wrong PIN, or the HSM client library can’t see the device. For SoftHSM in a lab, confirm the token was initialized and the SOFTHSM2_CONF path is right for the barbican user. The fix is never in Barbican’s secret logic — it’s in the HSM connection.

Step 4: When Octavia or Cinder can’t get their secret

Most real-world Barbican tickets come in disguised as Octavia or Cinder failures. Octavia stores its amphora TLS certs in Barbican; Cinder volume encryption stores LUKS keys there. When those services fail, the root cause is often a Barbican ACL or Keystone trust problem, not the service itself. Confirm the consuming service can actually read the secret:

openstack acl get <secret-href>
openstack secret container list

A 403 when Octavia tries to read its cert means the secret’s ACL doesn’t grant the service user — Barbican secrets are owned by a project and not automatically readable by service accounts unless you’ve granted access. This is the single most common cross-service Barbican failure.

Step 5: Lifecycle and expiration

Secrets can carry an expiration, and an expired secret stops being retrievable — which surprises people when a TLS cert’s Barbican entry expires before the cert does. Audit expirations and consumers before they bite:

openstack secret list --format value -c "Secret href" -c Expiration

Track which secrets have consumers (openstack secret container and the ACLs) so you never delete a secret an active load balancer or encrypted volume still depends on.

Where AI helps

Barbican failures are scary precisely because the stakes are high and the error messages (CKR_*, generic decrypt failures) are cryptic. I’ll paste the barbican-api log tail and the failing service’s error into a model and ask it to classify the failure: KEK/HSM unreachable, ACL/permission, expiration, or a genuine backend outage — and crucially, whether the problem risks existing secrets or just new operations. That distinction tells me whether I have an outage or a data-loss incident.

Keep a saved Barbican triage prompt and read our other OpenStack guides — especially Octavia and Cinder, since that’s where Barbican failures usually surface. The model reads the errors and reasons about blast radius; you run every command and you never, ever let it suggest rotating a KEK without a re-encryption plan.

Generated commands and configs are assistive, not authoritative. Always verify against your own deployment before running anything in production — and back up your KEK first.