Kolla-Ansible Certificate Update: Step-by-Step

Kolla-Ansible exposes three TLS surfaces, and knowing which one you are rotating is half the job. The external VIP (kolla_external_fqdn_cert) is the public endpoint that users, the OpenStack CLI, and Horizon connect to. The internal VIP (kolla_internal_fqdn_cert) carries service-to-service API traffic behind the control plane. And backend TLS (backend_tls) encrypts the last hop from HAProxy to each service container, with the CA copied into the containers so they can validate each other. In a standard Kolla-Ansible deployment, HAProxy terminates TLS on both VIPs — so the cert a client sees is whatever HAProxy is serving, not what any individual API container holds.

The certificate files live under /etc/kolla/certificates/ on the deploy host and are rendered into the haproxy container when you run a reconfigure. That is the key operational fact for a kolla ansible certificate update: you place new PEMs on the deploy host and run reconfigure — you do not deploy, and you rarely need to touch anything beyond HAProxy. This is the same TLS layer that, when it expires, produces the gateway errors covered in our OpenStack 504 Gateway Timeout guide, so getting rotation right is also outage prevention. If you are standing up TLS for the first time, our walkthrough on building a production-ready OpenStack cloud covers the initial layout.

Signs your certs need updating

You are probably here because you are seeing one or more of these:

• Browser or OpenStack CLI errors: certificate has expired or certificate verify failed when hitting the public endpoint.
• SSLError / SSL: CERTIFICATE_VERIFY_FAILED from one service calling another over the internal VIP — a symptom that the internal cert or its CA is stale.
• HAProxy is still serving an old certificate even after you copied a new file in (because you have not reconfigured yet).
• A monitoring alert warning of upcoming expiry on the external or internal VIP.
• Clients that previously worked now fail after a private CA was rotated but not copied into the containers.

Note that a TLS handshake failure between OpenStack and its message bus looks different — if RabbitMQ is involved, see RabbitMQ TLS handshake failed and RabbitMQ certificate verify failed, which are driven by the message-queue certs rather than the HAProxy VIP certs handled here.

Certificate locations & TLS concepts

Everything for a Kolla-Ansible TLS rotation lives in two places: the PEM files under /etc/kolla/certificates/, and the switches in /etc/kolla/globals.yml. HAProxy loads a combined PEM — server certificate, intermediate chain, and private key concatenated into one file — and serves it on the VIP. Kolla uses a fixed set of filenames:

The relevant globals.yml variables that control how those files are used:

• kolla_enable_tls_external: "yes" and kolla_enable_tls_internal: "yes" — turn TLS on for each VIP.
• kolla_external_fqdn_cert — path to the external combined PEM (defaults to haproxy.pem).
• kolla_internal_fqdn_cert — path to the internal combined PEM (defaults to haproxy-internal.pem).
• kolla_copy_ca_into_containers: "yes" — copies your root CA into every container's trust store so internal TLS validates. This is what makes a CA change a --tags common operation.
• kolla_admin_openrc_cacert — the CA path baked into the generated admin-openrc.sh, so the CLI trusts the endpoints.

The mental model: HAProxy is the only thing serving the VIP cert. Update the combined PEM, reconfigure HAProxy, and the new cert is live. The CA plumbing (kolla_copy_ca_into_containers) only matters when the issuer changes — a routine renewal from the same CA does not require touching every container.

Immediate checks (inspect current certs)

Before changing anything, establish exactly what is being served today and when it expires. Read the live endpoints and the on-disk PEMs — read-only, no writes:

# Point at your VIPs (no braces — plain shell vars)
VIP=your-external-vip
IVIP=your-internal-vip

# What the external VIP serves right now: dates + subject + SANs
openssl s_client -connect $VIP:443 -servername $VIP </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -ext subjectAltName

# Same for the internal VIP (services talk to this one)
openssl s_client -connect $IVIP:443 -servername $IVIP </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject

notAfter tells you the expiry; the subject/SAN must include the VIP FQDNs clients use, or verification fails even with a valid, unexpired cert.

# Expiry of the staged external and internal certs
openssl x509 -in /etc/kolla/certificates/haproxy.pem -noout -enddate -subject
openssl x509 -in /etc/kolla/certificates/haproxy-internal.pem -noout -enddate -subject

# Confirm the private key in haproxy.pem matches the cert (modulus must match)
openssl x509 -in /etc/kolla/certificates/haproxy.pem -noout -modulus | openssl md5
openssl rsa  -in /etc/kolla/certificates/haproxy.pem -noout -modulus | openssl md5

If the two md5 sums differ, the cert and key in the PEM do not match and HAProxy will fail to start — fix this before reconfiguring.

Safe backup steps

# Full, attribute-preserving copy of the certs dir (keep the old haproxy.pem to roll back)
cp -a /etc/kolla/certificates /etc/kolla/certificates.bak.$(date +%F)

# Also snapshot globals + passwords so a rollback is complete
cp -a /etc/kolla/globals.yml   /etc/kolla/globals.yml.bak.$(date +%F)
cp -a /etc/kolla/passwords.yml /etc/kolla/passwords.yml.bak.$(date +%F)

# Sanity-check the backup exists and holds the current PEMs
ls -l /etc/kolla/certificates.bak.$(date +%F)/

cp -a preserves ownership and permissions — important because HAProxy is picky about key file modes. Keep at least the previous haproxy.pem until the new cert is validated.

Update the certificates (reconfigure)

With a verified backup in place, stage the new certificates. For a lab or self-signed setup you can let Kolla generate them:

# Generates self-signed haproxy.pem / haproxy-internal.pem + a root CA into
# /etc/kolla/certificates/ for non-production testing only
kolla-ansible -i /etc/kolla/multinode certificates

Never use kolla-ansible certificates for production endpoints — clients won't trust the self-signed CA. Use a real CA or ACME for anything user-facing.

For real certificates, assemble the combined PEM by hand: the leaf server certificate first, then any intermediate chain, then the private key. Do the same for the internal PEM. HAProxy reads them as a single file, so ordering matters (leaf → chain → key):

# External VIP: server cert, then intermediate chain, then the private key
cat new-external.crt intermediate-chain.crt new-external.key \
  > /etc/kolla/certificates/haproxy.pem

# Internal VIP: same order with the internal cert/key
cat new-internal.crt intermediate-chain.crt new-internal.key \
  > /etc/kolla/certificates/haproxy-internal.pem

# If the issuing CA changed, refresh the root the containers trust
cp new-root-ca.crt /etc/kolla/certificates/ca/root.crt

# Lock down the private material and re-verify cert/key still match
chmod 600 /etc/kolla/certificates/haproxy.pem /etc/kolla/certificates/haproxy-internal.pem
openssl x509 -in /etc/kolla/certificates/haproxy.pem -noout -enddate -subject

Re-run the modulus md5 check from Immediate checks on the freshly built PEM before reconfiguring — a mismatched key here is the most common cause of a failed HAProxy reload.

Now reconfigure only the services that need it. For a straight cert renewal from the same CA, that is HAProxy alone — limiting --tags keeps Kolla from re-templating and restarting every container, which is what a full reconfigure would do:

# Same-CA renewal: HAProxy is the only surface that changed
kolla-ansible -i /etc/kolla/multinode reconfigure --tags haproxy

# CA changed too? Also re-copy the CA into all containers so internal TLS validates
kolla-ansible -i /etc/kolla/multinode reconfigure --tags common

# If public endpoints/SANs or the CA changed, Keystone catalog + Horizon may need a pass
kolla-ansible -i /etc/kolla/multinode reconfigure --tags keystone,horizon

Scoping --tags to haproxy (and common only when the CA changed) turns a 40-minute full reconfigure into a targeted 2-3 minute reload. Run the full reconfigure only if you genuinely changed cross-service TLS.

If you are folding this rotation into a larger maintenance event, sequence it the way you would any control-plane change — our guide on planning OpenStack upgrades safely covers windowing and rollback discipline that applies directly here.

Validation with curl / openssl / OpenStack CLI

Do not declare the rotation done because HAProxy came back up. Confirm the new cert is served, that it verifies without -k, and that services still talk over the internal VIP:

VIP=your-external-vip
IVIP=your-internal-vip

# 1) The served dates must match the NEW cert (compare to the enddate you staged)
openssl s_client -connect $VIP:443 -servername $VIP </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject

# 2) curl without -k must succeed (real trust chain, not skipped verification)
curl -sS -o /dev/null -w "external: %{http_code}\n" https://$VIP:5000/v3
curl -sS -o /dev/null -w "internal: %{http_code}\n" https://$IVIP:5000/v3

# 3) The internal VIP must serve the new internal cert too
openssl s_client -connect $IVIP:443 -servername $IVIP </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

If curl succeeds without -k, the chain and SANs are correct. A -k-only success means clients still won't trust it — fix the chain or CA before closing the window.

# Token issue exercises the full CLI -> HAProxy -> Keystone TLS path
openstack --os-cacert /etc/kolla/certificates/ca/root.crt token issue -f value -c id

# Services healthy over the internal VIP (these calls travel over internal TLS)
openstack endpoint list -f value -c "Service Name" -c URL | sort -u
openstack compute service list -f value -c Binary -c Status | sort -u

A clean token issue and endpoint list means both the external CLI path and internal service TLS trust the new cert. Errors like certificate verify failed here point at a missing chain or an un-copied CA.

Prevention & rollback

Rollback is the reason you took a backup. If the new cert is wrong — mismatched key, missing chain, wrong SANs, or clients refusing to trust it — restore the previous certificates directory and reconfigure HAProxy again:

# Restore the pre-change certs (use the backup timestamp you created)
cp -a /etc/kolla/certificates.bak.$(date +%F)/. /etc/kolla/certificates/

# Reload HAProxy with the restored cert
kolla-ansible -i /etc/kolla/multinode reconfigure --tags haproxy

# Verify the OLD cert is being served again
openssl s_client -connect your-external-vip:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject

Restore, reconfigure haproxy, verify — the same three steps as the forward path, just pointed at the backup. Keep the backup until the new cert is fully validated in production.

Prevention keeps you out of an expiry-driven incident entirely:

• Monitor cert expiry and alert 30 days out. Watch notAfter on both the external and internal VIPs; a browser error should never be your first signal. Wire this into the same stack you use for the rest of the control plane.
• Automate rotation with a proper CA or ACME (e.g. an internal step-ca or Let's Encrypt for public endpoints) so renewal is a scheduled job, not a manual scramble.
• Keep internal and external in sync. Rotate both VIPs together so service-to-service TLS never drifts against a stale CA — the mismatch that causes silent SSLErrors between services.
• Document the VIP FQDNs and required SANs so every reissue carries the right subject alternative names; a valid cert with wrong SANs still fails verification.
• Version-control globals.yml and treat cert paths as reviewed config, so a rotation is auditable and repeatable.

Want the prompts and tooling behind this workflow? Browse the AI prompt library for copy-paste Kolla-Ansible and TLS prompts, the free in-browser DevOps tools, and — when a production TLS incident needs senior hands — draft triage fast with the free AI Incident Response assistant or work with me directly.