🚨 Kubernetes Error Guide: 'remote error: tls: handshake failure' Protocol and Cipher Mismatch

Exact Error Message

tls: handshake failure occurs when two TLS peers cannot agree on the parameters needed to even begin a secure session. It shows up calling the API server, an admission webhook, or an aggregated API service:

$ kubectl get pods
Unable to connect to the server: remote error: tls: handshake failure

From the API server failing to reach a webhook:

W0628 17:22:10.553  1 dispatcher.go:142] Failed calling webhook validating.example.com:
  Post "https://webhook.svc:443/validate?timeout=10s": remote error: tls: handshake failure

A direct probe makes the negotiation failure explicit:

$ openssl s_client -connect webhook.svc:443
139844:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

What the Error Means

A TLS handshake begins with the client sending a ClientHello advertising its highest protocol version, a list of cipher suites, and (usually) an SNI server name. The server picks a mutually supported version and cipher. tls: handshake failure means that negotiation failed before any certificate was exchanged — there was no common protocol version or cipher suite, the server demanded SNI that was missing, or the server required a client certificate the client did not offer.

This is fundamentally different from a certificate-trust error. With handshake failure the peers never got far enough to validate identities; with x509: certificate signed by unknown authority or tls: bad certificate they did exchange certs and one was rejected. So the fix here is about TLS parameters, not trust chains.

Common Causes

• Protocol version mismatch — server enforces TLS 1.2/1.3 minimum while the client only offers TLS 1.0/1.1 (or a hardened server disabled the only version the client supports).
• No common cipher suite — server restricted to a cipher list the client cannot speak (e.g. TLS 1.3-only server vs older client).
• Missing or wrong SNI — server selects a certificate by SNI and rejects connections without it, or with an unrecognised server name.
• Mutual TLS expected — server requires a client certificate; a client offering none aborts with handshake failure.
• Webhook/aggregated API serving misconfig — TLS serving cert wired to the wrong port, or the service speaks plaintext on an HTTPS port.
• Hardened crypto policy / FIPS disabling algorithms one side needs.

How to Reproduce the Error

Stand up a server that only accepts TLS 1.3 and connect with a client capped at TLS 1.2:

# Server restricted to TLS 1.3 only (test endpoint)
openssl s_server -accept 8443 -cert s.crt -key s.key -tls1_3 &

# Client forced to TLS 1.2 -> no common protocol
openssl s_client -connect localhost:8443 -tls1_2
# -> sslv3 alert handshake failure

Because the client’s maximum (1.2) is below the server’s minimum (1.3), no version is shared and the server aborts the handshake before certificates are ever sent.

Diagnostic Commands

# Probe which protocols/ciphers the endpoint actually accepts (read-only)
openssl s_client -connect <HOST>:<PORT> -tls1_2 </dev/null
openssl s_client -connect <HOST>:<PORT> -tls1_3 </dev/null

# Test with explicit SNI to rule out an SNI requirement
openssl s_client -connect <HOST>:<PORT> -servername <SNI_HOST> </dev/null

# Does the server demand a client cert? (look for "Acceptable client certificate CA names")
openssl s_client -connect <HOST>:<PORT> -servername <SNI_HOST> -msg </dev/null | grep -i 'CertificateRequest\|client certificate'

# For a webhook: confirm port and serving config
kubectl get validatingwebhookconfiguration <NAME> -o jsonpath='{.webhooks[*].clientConfig.service}'; echo
kubectl get svc <WEBHOOK_SVC> -o jsonpath='{.spec.ports}'; echo

# Apiserver-side handshake errors
journalctl -u kube-apiserver --no-pager -n 50 | grep -i 'handshake'

Trying -tls1_2 then -tls1_3, with and without -servername, isolates whether the failure is a version, cipher, or SNI problem in a single sweep.

Step-by-Step Resolution

1. Pin down the failing parameter. Use the openssl s_client probes above. If -tls1_3 succeeds but -tls1_2 fails, it is a version floor. If both fail without -servername but succeed with it, it is an SNI requirement. If the server log mentions a client certificate request, it is mTLS.

2. Fix protocol/cipher mismatch. Align the client and server on a common TLS version and cipher suite. Either lower the server’s minimum to include what the client supports, or (preferably) upgrade the client. Hardened servers that disable TLS 1.2 will reject older clients until those clients are updated.

3. Supply SNI. If the server selects its certificate by SNI, ensure clients send the correct server name. For webhooks and aggregated APIs, the clientConfig.service name must match the cert’s SAN so SNI resolves.

4. Resolve mTLS expectations. If the server requires a client certificate, configure the client to present one (and verify it is trusted). If mTLS was not intended, disable the client-cert requirement on the server.

5. Check webhook/aggregated serving config. Confirm the webhook serves TLS on the targeted port and is not accidentally speaking plaintext. A service exposing an HTTP port where the apiserver expects HTTPS produces handshake failures.

6. Re-probe to confirm. Re-run openssl s_client; a completed handshake (Verify return code line appears) confirms negotiation now succeeds.

Prevention and Best Practices

• Standardise a supported TLS version floor (TLS 1.2+ ) across clients and servers, and roll version/cipher hardening out to clients before tightening servers.
• Ensure every TLS serving cert includes the SNI host(s) as Subject Alternative Names and that clients send SNI.
• Document whether each endpoint expects mutual TLS so clients are configured with the right client cert (or none) up front.
• Monitor for handshake failures in apiserver and webhook logs; a spike after a security upgrade signals a version/cipher regression.
• Test webhook and aggregated API serving with openssl s_client in CI before rollout.
• Keep crypto policies (FIPS, system-wide hardening) consistent across the fleet so one node does not silently drop a needed cipher. More in our Kubernetes & Helm guides.

Related Errors

• remote error: tls: bad certificate — the client cert was exchanged and rejected (later stage than handshake failure).
• x509 certificate signed by unknown authority — the server cert is distrusted after a successful handshake.

Frequently Asked Questions

How is handshake failure different from bad certificate? Timing. handshake failure happens during ClientHello/ServerHello negotiation — before certificates are exchanged — so it is about protocol versions, ciphers, or SNI. bad certificate happens after certificates are exchanged, when one side rejects the other’s cert. If you never see a cert in the trace, it is a handshake-parameter problem.

It started failing right after a Kubernetes or OS upgrade. Newer releases routinely raise the minimum TLS version and prune weak ciphers. An older client or webhook that only offered TLS 1.0/1.1 or a now-disabled cipher suddenly shares nothing with the upgraded server. Update the lagging side to a modern TLS version.

Why does openssl s_client work but my application fails the handshake? Often SNI or client-cert differences. openssl may be sending a server name (or omitting a client cert) that differs from your app’s behaviour. Re-run with -servername set to exactly what your app sends, and add -cert/-key if the server requires mTLS, to reproduce the app’s path.

A webhook gives handshake failure but the pod is healthy. Check that the webhook actually serves TLS on the port the API server targets and that SNI matches the serving cert’s SAN. A webhook listening with plaintext on an HTTPS port, or a cert without the service name in its SAN, fails the handshake even though the pod is Running.