🚨 Kubernetes Error Guide: 'failed to rotate certificate' Pending Kubelet CSR

Exact Error Message

# journalctl -u kubelet on the node
kubelet: E0627 13:22:41 certificate_manager.go: Failed while requesting a signed
  certificate from the control plane: cannot create certificate signing request:
  Unauthorized
kubelet: E0627 13:22:41 kubelet_node_status.go: Error updating node status,
  will retry: error getting node "worker-3": Unauthorized

# kubectl get csr
NAME        AGE   SIGNERNAME                       REQUESTOR                 CONDITION
csr-7g4k2   9m    kubernetes.io/kubelet-serving    system:node:worker-3      Pending
csr-pq8sd   9m    kubernetes.io/kube-apiserver-client-kubelet  system:bootstrap:abc  Pending

What the Error Means

The kubelet authenticates to the API server with a client certificate. To avoid long-lived credentials, the kubelet rotates this certificate automatically: before expiry, it generates a new key and submits a CertificateSigningRequest (CSR) to the control plane, waits for it to be approved and signed, then swaps in the new certificate. Rotation has two flavors — the client certificate (kubernetes.io/kube-apiserver-client-kubelet) used to talk to the API server, and the serving certificate (kubernetes.io/kubelet-serving) the kubelet presents for kubectl logs/exec.

Failed while requesting a signed certificate / Unauthorized, or a CSR stuck in Pending, means rotation did not complete. If the existing client cert then expires, the kubelet can no longer authenticate at all, the node goes NotReady, and pods on it become unreachable. This is a slow-burn failure that often surfaces exactly one year after cluster creation (the default cert lifetime).

Common Causes

• CSRs are not being approved because the cluster lacks (or has misconfigured) an auto-approver, so requests sit Pending.
• The kubelet client certificate already expired, so the kubelet cannot even submit a new CSR (Unauthorized), creating a chicken-and-egg lockout.
• Clock skew between node and control plane, making the new cert appear not-yet-valid or already expired.
• The bootstrap token expired, so a (re)joining node cannot authenticate to create its initial CSR.
• RBAC missing for system:node or the bootstrapper to create/approve CSRs.
• rotateCertificates disabled in the kubelet config, so no rotation is attempted and the cert simply expires.

How to Reproduce the Error

In a lab cluster, the simplest trigger is to let (or force) the kubelet client cert to expire while CSR auto-approval is off:

# Disable the auto-approver-equivalent RBAC (lab only), then on a node:
sudo rm /var/lib/kubelet/pki/kubelet-client-current.pem
sudo systemctl restart kubelet
# The kubelet submits a CSR that stays Pending; the node goes NotReady.
kubectl get csr
kubectl get nodes

Diagnostic Commands

# List CSRs and their state (look for Pending)
kubectl get csr
kubectl get csr -o wide

# Inspect a specific CSR: signer, requestor, usages
kubectl describe csr <CSR_NAME>

# Node readiness and last heartbeat
kubectl get nodes
kubectl describe node <NODE> | grep -A6 -iE 'conditions|lease'

# On the node: kubelet cert errors and expiry
journalctl -u kubelet --since "30 min ago" | grep -iE 'certificate|csr|unauthorized'
sudo openssl x509 -enddate -noout -in /var/lib/kubelet/pki/kubelet-client-current.pem

# Control-plane cert expiry overview (kubeadm clusters)
sudo kubeadm certs check-expiration

# Verify clocks are in sync (skew breaks cert validity)
timedatectl status

Step-by-Step Resolution

1. Find the Pending CSRs. Identify which signer is stuck — client (...client-kubelet) or serving (kubelet-serving):

kubectl get csr

2. Approve the pending CSRs. If they are legitimate node requests, approve them so the cert is signed and the kubelet can complete rotation:

kubectl certificate approve <CSR_NAME>
# Approve all pending at once (verify they are expected first):
kubectl get csr -o name | xargs -r kubectl certificate approve

Within seconds the kubelet picks up the signed cert and the node should return to Ready.

3. If the client cert already expired (Unauthorized), re-bootstrap the kubelet. The kubelet cannot self-submit once locked out. On a kubeadm node, generate a fresh bootstrap kubeconfig and restart:

# On the control plane: create a new bootstrap token
sudo kubeadm token create --print-join-command
# On the affected node: refresh /etc/kubernetes/bootstrap-kubelet.conf with the token,
# remove the stale client cert, and restart the kubelet
sudo rm -f /var/lib/kubelet/pki/kubelet-client-current.pem
sudo systemctl restart kubelet

The kubelet then submits a new CSR using the bootstrap token; approve it as in step 2.

4. Restore CSR auto-approval. Ensure the cluster auto-approves node client CSRs so this does not recur. Confirm the relevant ClusterRoleBindings for system:certificates.k8s.io:certificatesigningrequests:nodeclient exist (managed automatically on kubeadm clusters).

5. Fix clock skew if present. Sync NTP on the node so signed certs are accepted:

sudo timedatectl set-ntp true

6. Confirm recovery. The node should be Ready and the new cert should have a future expiry:

kubectl get nodes
sudo openssl x509 -enddate -noout -in /var/lib/kubelet/pki/kubelet-client-current.pem

Prevention and Best Practices

• Keep rotateCertificates: true (and serving-cert rotation) enabled in the kubelet configuration so renewal happens automatically before expiry.
• Ensure CSR auto-approval is configured for node client certificates so rotation does not depend on a human approving CSRs.
• Monitor certificate expiry proactively — alert weeks ahead with kubeadm certs check-expiration and node-level cert checks.
• Keep NTP enabled on every node and control-plane host to avoid clock-skew rejections.
• Renew bootstrap tokens or use a non-expiring join mechanism for nodes that may rejoin after long downtime.
• For kubeadm control planes, renew control-plane certs with kubeadm certs renew all well before the one-year default. See more in Kubernetes & Helm guides.

Related Errors

• Unauthorized from the API server — the kubelet’s expired/invalid client cert was rejected.
• x509: certificate has expired or is not yet valid — clock skew or an expired cert in the chain.
• NodeNotReady — the downstream node state when the kubelet can no longer authenticate.
• x509: certificate signed by unknown authority — a CA/trust mismatch rather than an expiry/rotation issue.

Frequently Asked Questions

Why did all my nodes go NotReady around the same time, roughly a year in? The default kubelet (and control-plane) certificate lifetime is about one year. If rotation or auto-approval was not working, every node’s cert expires around its creation anniversary, taking nodes NotReady together. Approve the pending CSRs and fix auto-approval.

The kubelet says Unauthorized and cannot create a CSR. How do I recover? The existing cert expired, so the kubelet has no valid identity to request a new one. Re-bootstrap it: create a fresh bootstrap token, refresh the bootstrap kubeconfig on the node, remove the stale client cert, and restart the kubelet so it submits a new CSR.

Is it safe to approve all pending CSRs at once? Only after verifying they are legitimate node requests (correct signer and system:node:/system:bootstrap: requestor). Blindly approving unknown CSRs is a security risk. Inspect with kubectl describe csr first.

Does approving a CSR restart my pods? No. Approving a CSR only signs the kubelet’s certificate. Once the node re-authenticates and returns to Ready, its existing pods continue running; nothing is rescheduled by the approval itself.