🚨 mTLS Error: 'tls: bad certificate' Client Authentication Rejected

Overview

In mutual TLS (mTLS) the server requires the client to present a certificate and validates it against a trusted CA. When the client’s certificate is missing, untrusted, expired, or malformed, the server aborts the handshake and both sides log a TLS alert. On a Go service the server logs:

http: TLS handshake error from 10.0.4.11:53122: remote error: tls: bad certificate

When the client sent nothing at all and the server is set to require verification, the alert is more specific:

http: TLS handshake error from 10.0.4.11:53140: tls: client didn't provide a certificate

From the client’s side (curl against an mTLS endpoint) the same rejection reads:

curl: (56) OpenSSL SSL_read: error:0A00045C:SSL routines::tlsv13 alert certificate required

This is an authentication failure at the transport layer. Encryption negotiation started, but the server refused to complete the handshake because the client identity did not satisfy policy. No HTTP request is ever processed.

Symptoms

• A service-to-service call fails immediately with remote error: tls: bad certificate and no application log entry, because the request never reached the handler.
• curl returns (56) ... alert certificate required or (58) unable to set private key file when calling an mTLS endpoint.
• A sidecar/mesh proxy (Envoy, Istio) reports TLS_error / certificate_verify_failed and upstream health checks flap.
• Calls work from one client but fail from another that was issued a cert from a different CA.
• Everything worked until a certificate rotation or CA renewal, then all clients started failing at once.

Common Root Causes

• Client sent no certificate — the client was not configured with a cert/key pair, or a proxy stripped it, while the server is set to RequireAndVerifyClientCert.
• Untrusted issuing CA — the client cert is valid but signed by a CA that is not in the server’s client-CA trust bundle (ClientCAs), so verification fails.
• Expired or not-yet-valid cert — the client leaf or an intermediate in its chain is outside its notBefore/notAfter window.
• Incomplete chain — the client presents only its leaf and omits the intermediate, so the server cannot build a path to the trusted root.
• Key usage / EKU mismatch — the cert lacks clientAuth extended key usage, so it is rejected for client authentication even though it works as a server cert.
• Wrong CA bundle after rotation — the server’s ClientCAs still points at the old CA, or the new CA was added to the wrong side.
• Revocation — the cert was revoked and the server enforces CRL/OCSP.
• Identity mismatch in the app layer — the handshake succeeds but the server then rejects the cert’s SAN/SPIFFE ID against an authorization policy (a related but distinct failure worth ruling out).

Diagnostic Workflow

Reproduce the handshake with openssl s_client, presenting the client cert and printing the verification result:

openssl s_client -connect api.internal:8443 \
  -cert client.crt -key client.key \
  -CAfile server-ca.pem -state 2>&1 | head -40

Inspect the client certificate’s validity window, issuer, SANs, and extended key usage:

openssl x509 -in client.crt -noout -dates -issuer -subject
openssl x509 -in client.crt -noout -ext extendedKeyUsage,subjectAltName

Confirm the client chain actually builds to the CA the server trusts:

openssl verify -CAfile server-client-ca.pem \
  -untrusted intermediate.crt client.crt

Check what the server was actually sent and whether it asked for a cert. Turn up Go/keylog or read the proxy access log; with curl, use verbose:

curl -v --cert client.crt --key client.key \
  --cacert server-ca.pem https://api.internal:8443/health

Verify the server’s trust bundle contains the client’s issuing CA:

openssl crl2pkcs7 -nocrl -certfile /etc/tls/client-ca-bundle.pem \
  | openssl pkcs7 -print_certs -noout | grep -i 'subject\|issuer'

For a service mesh, check the proxy’s certificate config and secret state:

istioctl proxy-config secret <pod> -n <ns>          # Istio
openssl s_client -connect <upstream> -showcerts      # confirm chain

Example Root Cause Analysis

After a routine renewal of an internal issuing CA, a fleet of payment workers began failing every call with remote error: tls: bad certificate, while the servers logged nothing at the application layer. openssl s_client from a worker showed verify error:num=20:unable to get local issuer certificate and the presented chain terminating at the new intermediate.

The new client certificates chained to a renewed intermediate, but the servers’ ClientCAs bundle still contained only the old intermediate. Because verification builds a path from the presented leaf to a trusted anchor, and the anchor for the new intermediate was absent, every handshake was correctly rejected. The fix was to add the new intermediate to the server-side client-CA bundle and reload — keeping the old one during overlap so in-flight certs from both generations verified:

cat intermediate-old.pem intermediate-new.pem root.pem \
  > /etc/tls/client-ca-bundle.pem
sudo systemctl reload api-gateway

Handshakes recovered immediately. The unsafe shortcut, switching the server to RequireAnyClientCert (skip verification) or InsecureSkipVerify, would have “worked” while accepting any self-signed cert and destroying the entire point of mTLS.

Prevention Best Practices

• Distribute CA bundles before rotating leaf certs. Add the new issuing CA to every server’s client-trust bundle first, keep the old one during an overlap window, then remove it — never flip both at once.
• Ship full chains. Configure clients to present leaf plus intermediates so the server can always build a path; do not rely on the server having intermediates.
• Set clientAuth EKU on client certs. Issue distinct client and server profiles so a cert cannot be misused across roles.
• Automate issuance and short TTLs. Use cert-manager, Vault PKI, or SPIFFE/SPIRE to rotate short-lived client certs automatically, with expiry monitoring and alerting before notAfter.
• Keep verification strict. Use RequireAndVerifyClientCert (Go) / ssl_verify_client on (nginx); never disable verification or set InsecureSkipVerify to clear an error.
• Enforce revocation with OCSP stapling or short lifetimes so a compromised client cert can be withdrawn.
• Separate identity from transport. Authorize on the validated SAN/SPIFFE ID at the app layer, so a valid cert still only reaches what its identity is allowed to.

Quick Command Reference

# Reproduce and inspect the handshake
openssl s_client -connect host:8443 -cert client.crt \
  -key client.key -CAfile server-ca.pem -state

# Client cert details: dates, issuer, SAN, EKU
openssl x509 -in client.crt -noout -dates -issuer -subject
openssl x509 -in client.crt -noout -ext extendedKeyUsage,subjectAltName

# Does the client chain verify to the trusted CA?
openssl verify -CAfile client-ca.pem -untrusted intermediate.crt client.crt

# What does curl see end to end?
curl -v --cert client.crt --key client.key --cacert server-ca.pem \
  https://host:8443/health

# List subjects in the server's client-CA trust bundle
openssl crl2pkcs7 -nocrl -certfile client-ca-bundle.pem \
  | openssl pkcs7 -print_certs -noout | grep subject

Conclusion

tls: bad certificate in an mTLS setup means the server did exactly what it was told: it demanded a client identity and rejected one it could not trust. Diagnose it as a chain-and-trust problem — reproduce with openssl s_client, check the client cert’s dates, issuer, chain, and clientAuth EKU, then confirm the server’s ClientCAs bundle actually anchors the client’s issuing CA. The most common trigger is a CA rotation where trust bundles and leaf certs fell out of sync, fixed by distributing the new CA with an overlap window. Whatever the cause, never resolve it by disabling verification; strict mutual verification is the security property you deployed mTLS to get.