⎈ The Certificate Nobody Owned: A Kubernetes TLS War Story

Troubleshooting Kubernetes · Part 4 — start from Part 1 or see the full series →

Bring coffee, I said at the end of the last one. Bring the runbook. Bring, if you can find it, whoever set the renewal reminder.

There was no runbook. There was no renewal reminder. And there was no whoever — which, it turned out, was the entire problem. It was 3:47 on a Sunday morning, the kind of hour that only exists in incident timelines and regret, and an entire Kubernetes cluster had quietly stopped doing the one thing a cluster is for: changing.

Welcome back to Troubleshooting Kubernetes. This is the one about a single expired certificate that nobody owned, that took down not a service but the ability to deploy any service — and about how, once again, absolutely nothing was broken. Two systems each did exactly what they were told, correctly, on schedule, and the sum of two correct behaviors was a frozen cluster. You’re going to like this one. I did, eventually. At about 5 a.m.

The page that made no sense

The alert was “deployments not progressing,” which is a strange thing to be paged for, because it’s not an outage in the usual sense. The apps were all up. Traffic was serving. Nothing was throwing 500s. By every customer-facing measure the cluster was fine.

But nothing could change. A deploy that had kicked off at 3:40 was still sitting there, pods untouched. The HorizontalPodAutoscaler wanted to scale up for the Sunday-morning batch load and couldn’t. Cert-manager, ironically, couldn’t renew anything. Nothing was reconciling. The cluster was frozen in amber — perfectly alive, utterly unable to move.

First instinct, the one you have at 3:47 with your eyes half open: is the API server down? I ran the most basic thing there is:

kubectl get pods -A
# ...returns instantly. Everything Running. Everything fine.

Reads worked. Perfectly. So the API server was up. Which made “nothing can change” even more baffling — the front door was open, the lights were on, and every request to write something just… vanished into a hang and then an error.

The error that named the culprit

So I stopped reading and tried to write. The smallest, dumbest write I could think of:

kubectl label node node-1 canary=please-work --overwrite

And there it was, the whole story in one line:

Error from server (InternalError): Internal error occurred: failed calling
webhook "validate.policy.internal": failed to call webhook: Post
"https://policy-webhook.platform.svc:443/validate?timeout=10s": x509:
certificate has expired or is not yet valid: current time 2026-07-05T03:47:11Z
is after 2026-07-05T02:00:00Z

Read that last clause slowly, because it’s the entire incident: current time is after 2026-07-05T02:00:00Z. A certificate had expired at 2 a.m. And every write to the cluster was now dying trying to talk to the thing that certificate belonged to.

How one cert freezes everything

Here’s the mechanism, and it’s a beautiful, terrible piece of design working exactly as intended.

There was a validating admission webhook in the cluster — a policy engine that inspects every write to the API and says yes or no. Someone had installed it, a long time ago, to enforce rules: no privileged pods, required labels, that sort of thing. Sensible. Good hygiene.

An admission webhook is an HTTPS service, and the API server talks to it over TLS. Its ValidatingWebhookConfiguration had this field:

failurePolicy: Fail

failurePolicy: Fail means: if I cannot reach this webhook to ask whether a write is allowed, I will refuse the write. Fail closed. And that is the correct, safe setting — the whole point of a policy webhook is that you don’t want writes sneaking past it when it’s down. An Ignore policy would let anything through the moment the webhook hiccuped, which defeats the purpose.

So: the webhook’s serving certificate expired at 2 a.m. The TLS handshake between the API server and the webhook started failing with x509: certificate has expired. The API server, unable to verify any write was policy-compliant, did exactly what failurePolicy: Fail told it to do — it refused every write. All of them. Including the writes that controllers, the scheduler, cert-manager, and every deploy needed to make. Reads don’t go through admission, so they sailed through fine, which is why the cluster looked healthy while being completely paralyzed.

The API server wasn’t down. It was up, healthy, and faithfully rejecting every change because a safety policy told it to fail closed against a certificate that had faithfully expired on schedule. Two correct behaviors. One frozen cluster.

The question nobody could answer

I found the webhook. I found its expired cert. And then I asked the question that turned a technical incident into an organizational one:

Who owns this?

Nobody knew. It had been installed over a year ago by a platform initiative that had since been reorganized out of existence. The serving cert had a 365-day validity. There was a rotation CronJob, once — I found its corpse, a Job that had been failing silently for two months after a ServiceAccount it depended on got cleaned up in an unrelated tidy-up. No alert watched the cert’s expiry. No dashboard showed it. No human had looked at that certificate since the day it was minted, 365 days ago, which is exactly how long it was designed to last.

The cert didn’t fail. It succeeded — it lasted precisely as long as it promised to, and then it did the one thing certificates are built to do. Expire. The failure wasn’t the certificate. The failure was that it belonged to no one and nothing was watching the calendar.

Stopping the bleeding, then fixing it

At 4 a.m. you have two jobs, in order: unfreeze the cluster, then fix it properly.

Unfreeze it. The fastest safe move is to renew the webhook’s cert and update the caBundle the API server trusts. If the cert is managed by cert-manager, force it:

# regenerate the serving cert, then confirm the webhook's caBundle matches
kubectl delete certificaterequest -n platform --all   # let cert-manager reissue
kubectl get secret policy-webhook-tls -n platform -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

If renewal is going to take time and the cluster needs to move now, the emergency valve is to stop the webhook from failing closed — but understand exactly what you’re trading:

# EMERGENCY ONLY: let writes through unvalidated until the cert is fixed.
kubectl patch validatingwebhookconfiguration validate.policy.internal \
  --type=json -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'

That un-freezes the cluster instantly — because now the API server allows writes it can’t validate. Which means for as long as that patch is in place, your policy engine is off and unenforced. It’s the right call at 4 a.m. to restore change; it’s the wrong thing to leave in place, and you set an alarm to revert it the moment the cert is healthy. (Deleting the webhook config entirely does the same thing, more bluntly.)

Fix it for real. Three things, none of them the cert itself:

1. Automate rotation properly — cert-manager issuing and renewing the webhook cert, with the renewal wired to update the caBundle (a CA injector), so no human is ever the rotation mechanism.
2. Alert on expiry — a Prometheus rule on x509 cert expiry (blackbox exporter probing the endpoint, or a cert-expiry exporter) that pages at 30 days out, not at hour zero. You want a boring Tuesday warning, not a 3:47 Sunday page.
3. Give it an owner — every webhook, every cert, every failurePolicy: Fail gets a name attached to it. Fail-closed dependencies are load-bearing; a load-bearing thing that belongs to nobody is an outage with a countdown timer already running.

Why I still love this

I reverted the Ignore patch at 6:12 a.m. with a renewed cert in place and a Prometheus alert freshly wired to page us thirty days before any cert in that cluster goes near its expiry. Tidy ending. Not the part I think about.

The part I think about is that nothing in this incident was a bug. The certificate did exactly what a certificate does — it expired, on time, as designed. failurePolicy: Fail did exactly what it’s for — it refused to let unvalidated changes through. The API server did exactly what it was told. Every single component behaved with total integrity. And the result was a cluster frozen solid at 3:47 on a Sunday, because one expiry date in one certificate had drifted out of every human’s field of view and nobody had been assigned to keep it in sight.

That’s the whole series, really, in one incident. The machine is honest. It executes what it’s told — the policies, the expiry dates, the fail-closed safety rails — with a literalness that never wavers. It won’t save you from a decision nobody remembered making. Reading the floorboards, this time, meant reading a date field: Not After: 2026-07-05 02:00:00 UTC, sitting in a certificate that had been quietly counting down for 365 days while the team that minted it dissolved around it. The fix took ten minutes. The lesson — that a fail-closed dependency with no owner is just a scheduled outage — took the whole rest of the night, and it was, I’ll admit it, worth every minute. Twenty-five years in, a single expired cert can still teach me something at 5 a.m. and leave me grinning about it on the drive home.

Next time in Troubleshooting Kubernetes: the namespace that got stuck Terminating and would not, could not, go away — for three days — because something invisible was holding the door. Bring coffee. Bring kubectl get and a lot of patience. Bring, this time, a basic understanding of finalizers, which I very much did not have when it started.

A cert-expiry alert and an owner for every fail-closed dependency will save you more Sunday mornings than almost anything else in Kubernetes. The running set of Kubernetes runbooks and error guides I keep — so the next 3:47 a.m. starts from a known error and not a blank terminal — lives on the Kubernetes toolkit. Read the floorboards. And the expiry dates.