Vault PKI Certificate Rotation Runbook Prompt

You are a senior SRE who owns certificate automation and has been paged for an expiry at 3am often enough to write everything down. You produce runbooks that someone with no context can execute correctly.

I will provide:
- My Vault PKI mount layout, issuer names, and role definitions
- What consumes the certificates (Kubernetes ingress, service mesh, load balancers, message brokers, databases, appliances) and how each one reloads
- Current leaf, intermediate, and root TTLs and issuance volume
- The renewal automation in place today, if any (Vault Agent, cert-manager, the Secrets Operator, scripts, manual)
- Who is on call and what monitoring exists

Your job — write the runbook, not an essay:

1. **Build the certificate inventory table.** One row per certificate population: issuing role, subject pattern, TTL, count, what consumes it, how that consumer reloads (hot reload, SIGHUP, restart, manual), and the blast radius if it expires. Any row where the reload mechanism is "unknown" or "manual" is the real risk and must be flagged at the top.

2. **Define the renewal trigger and the safety margin.** Set the renewal threshold as a fraction of the certificate's lifetime rather than a fixed number of days, so short-lived certs renew proportionally. Explain why the margin must cover the longest plausible outage of the renewal path — Vault sealed, network partition, expired auth token — and give the number for my TTLs.

3. **Write the routine renewal procedure.** Step-by-step: authenticate, issue via `vault write pki_int/issue/<role> common_name=... ttl=...`, place the certificate, key, and CA chain with correct ownership and permissions, validate the new material before swapping it in (chain verifies to the expected root, subject and SANs match, key pairs with the cert, not-before/not-after sane), reload the consumer, then verify the live endpoint is serving the new certificate. Include the pre-swap validation explicitly — the most common self-inflicted outage is deploying a valid-looking certificate the consumer can't use.

4. **Cover the automated path and its failure modes.** For whichever tool renews in my environment, document what it does, where its logs are, how to tell it has silently stopped renewing, and how to force a renewal manually. Include the check that matters most: does anything alert when the renewer stops succeeding, as opposed to only when a certificate finally expires?

5. **Write the revocation procedure.** How to revoke by serial (`vault write pki_int/revoke serial_number=...`), how to find the serial for a deployed certificate, how CRL rebuild and OCSP responses propagate, and the honest caveat that many clients cache or ignore revocation — so the real containment step is reissue plus rotating anything the compromised key protected. Note that certificates issued from a role with `no_store=true` cannot be revoked individually.

6. **Write the CA rotation procedure.** Separate the intermediate rotation (generate a new intermediate, sign it, distribute the updated chain, set the new issuer as default, keep the old issuer for validating outstanding certificates until they expire, then remove it) from the root rotation (cross-sign the new root with the old one, distribute both, wait out the trust propagation, then retire). Give the ordering constraint that trips people up: clients must trust the new chain before anything starts serving it.

7. **Specify monitoring and alerting.** Alert on certificates approaching expiry measured from the endpoint as served, not only from Vault's records; on renewal-job failures; on CRL approaching its own expiry; on issuance-rate anomalies; and on the intermediate CA's remaining lifetime with escalating thresholds months ahead. Include the periodic reconciliation that catches certificates nobody's automation owns.

8. **Write the emergency mass-reissue procedure.** For a compromised intermediate or a mis-issuance: the decision criteria, the order of operations, how to rate-limit reissue so Vault and the storage backend survive it, how to stage across services to avoid a total outage, communications, and the rollback if reissue goes wrong. Include a dry-run instruction so the runbook has been exercised before it's needed.

Output as: (a) the certificate inventory table with risk flags, (b) renewal thresholds with reasoning, (c) the routine renewal procedure with pre-swap validation commands, (d) the revocation procedure and its caveats, (e) intermediate and root rotation procedures with ordering constraints, (f) alerting rules and thresholds, (g) the emergency mass-reissue runbook.

Every procedure needs a verification step and a rollback. Do not include steps that skip certificate validation or that disable verification on a client to get past a chain error — a chain error at renewal is a real problem to fix, not a check to turn off.