🚨 Kubernetes Error Guide: 'remote error: tls: bad certificate' Client Cert Rejected

Exact Error Message

tls: bad certificate appears during a mutual-TLS handshake when the server rejects the client’s certificate. You will see it from kubectl, the API server log, or an admission webhook:

$ kubectl get pods
Unable to connect to the server: remote error: tls: bad certificate

From the API server reaching out to a webhook whose client cert it distrusts:

W0628 16:40:02.118  1 dispatcher.go:142] Failed calling webhook validating.example.com:
  Post "https://webhook.svc:443/validate?timeout=10s": remote error: tls: bad certificate

And the server side recording the rejection:

http: TLS handshake error from 10.0.5.6:51022: remote error: tls: bad certificate

What the Error Means

This is a mutual TLS failure on the client certificate. In mTLS both sides present certificates: the server proves its identity (verified against the client’s CA bundle) and the client proves its identity (verified against the server’s --client-ca-file). remote error: tls: bad certificate means the server evaluated the client’s certificate and refused it — wrong issuing CA, expired, not yet valid, revoked, or wrong key usage.

The word “remote” matters: the error is reported by the side that received the alert. When kubectl prints it, the API server sent the alert because your client cert was rejected. When the apiserver log prints it about a webhook, the webhook rejected the apiserver’s client cert. The fix is always on the side whose certificate was rejected, validated against the rejecting side’s trust store.

Common Causes

• Client cert signed by the wrong CA — the kubeconfig/webhook client cert was issued by a CA not in the server’s --client-ca-file (or the webhook’s trust config).
• Expired or not-yet-valid client cert — kubelet/kubeconfig client certificate past notAfter (or clock skew making it notBefore).
• Wrong cert/key pair — client-certificate-data and client-key-data in the kubeconfig do not match.
• Missing client auth EKU — the cert lacks Extended Key Usage: clientAuth.
• Rotated CA without distributing the new bundle — server CA changed but clients still present old-CA certs (or vice versa).
• Webhook caBundle / apiserver client cert mismatch in --admission-control-config-file.

How to Reproduce the Error

Build a client cert from a CA the API server does not trust and try to use it:

# Self-signed CA the cluster does NOT trust
openssl req -x509 -newkey rsa:2048 -nodes -keyout rogue-ca.key \
  -out rogue-ca.crt -days 1 -subj "/CN=rogue-ca"
# Client cert signed by that rogue CA, then point a kubeconfig at it...
kubectl --client-certificate=rogue.crt --client-key=rogue.key \
  --server=https://<APISERVER>:6443 get pods
# -> Unable to connect to the server: remote error: tls: bad certificate

The API server checks the presented client cert against --client-ca-file, finds the issuer untrusted, and aborts with bad certificate.

Diagnostic Commands

# Inspect your client cert: issuer, validity window, and EKU (read-only)
kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d | openssl x509 -noout -issuer -subject -dates -ext extendedKeyUsage

# Verify the client cert chains to the CA the server trusts
openssl verify -CAfile /etc/kubernetes/pki/ca.crt client.crt

# Confirm cert and key actually match (moduli must be identical)
openssl x509 -noout -modulus -in client.crt | openssl md5
openssl rsa  -noout -modulus -in client.key | openssl md5

# Watch the apiserver / webhook reject the handshake
journalctl -u kube-apiserver --no-pager -n 50 | grep -i 'tls handshake'

# For a webhook: what client cert does the apiserver present, and what CA does the webhook trust?
kubectl get validatingwebhookconfiguration <NAME> -o jsonpath='{.webhooks[*].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject -dates

The openssl verify result is decisive: if the client cert does not chain to the server’s CA file, that is exactly the rejection.

Step-by-Step Resolution

1. Identify which certificate was rejected. Determine the direction: kubectl printing it = your client cert; apiserver-log webhook warning = apiserver’s client cert to the webhook. Fix the rejected cert against the rejecting party’s trust store.

2. Check expiry and validity window. Run openssl x509 -noout -dates. An expired client cert (very common with kubelet/admin certs) produces exactly this error. Renew or rotate it.

3. Verify the issuing CA. Use openssl verify -CAfile <server-ca> client.crt. If it fails, the cert was signed by the wrong CA — reissue it from a CA present in the server’s --client-ca-file, or add the correct CA to that file.

4. Confirm the key matches the cert. Compare moduli with the two openssl ... -modulus | md5 commands. Mismatched cert/key pairs in a kubeconfig also trigger handshake failures.

5. Check key usage. The client cert must include Extended Key Usage: TLS Web Client Authentication. A server-only cert reused as a client cert is rejected.

6. For webhooks, align caBundle and client cert. Ensure the apiserver’s client cert (from its admission/egress config) is signed by a CA listed in the webhook’s expectations, and that the webhook’s clientConfig.caBundle matches its serving CA. Update whichever drifted.

Prevention and Best Practices

• Monitor client-certificate expiry (kubelet, admin kubeconfigs, webhook clients) and rotate well before notAfter — expiry is the most frequent cause.
• Prefer kubelet serving/client cert auto-rotation (--rotate-certificates) so node certs renew automatically.
• When rotating a CA, distribute the new bundle to every trusting party before retiring the old CA; keep both in --client-ca-file during the overlap.
• Keep clocks synced with NTP across control plane and nodes; skew makes valid certs look notBefore/expired.
• Treat webhook caBundle and the apiserver client cert as a managed pair (cert-manager CABundle injection helps).
• Always set clientAuth EKU on client certs. More in our Kubernetes & Helm guides.

Related Errors

• x509 certificate signed by unknown authority — the server cert is distrusted (opposite direction).
• remote error: tls: handshake failure — protocol/cipher/SNI mismatch before cert validation.

Frequently Asked Questions

Whose certificate is “bad” — mine or the server’s? The client’s. tls: bad certificate is sent by the side validating the client certificate in an mTLS handshake. If kubectl prints it, the API server rejected your client cert. The server’s own certificate being distrusted is a different error: x509: certificate signed by unknown authority.

My cert was working yesterday and broke overnight — why? Almost certainly expiry. Client certs (especially short-lived kubelet and admin certs) cross their notAfter and the server immediately rejects them. Run openssl x509 -noout -dates on the cert; renew or re-issue it.

A webhook started failing with bad certificate after I rotated a CA. The API server presents a client cert the webhook no longer trusts (or the webhook’s serving CA changed and the apiserver’s caBundle is stale). Re-sign the apiserver client cert from a trusted CA and update the webhook’s clientConfig.caBundle to the current serving CA.

openssl verify passes but I still get bad certificate. Check the Extended Key Usage and the cert/key pairing. A cert that chains correctly but lacks clientAuth EKU, or a kubeconfig where the key does not match the cert, both fail the handshake even though the chain is valid.