🚨 Security Error Guide: etcd 'remote error: tls: bad certificate' (mTLS)

Overview

etcd — the key-value store behind Kubernetes and many distributed systems — is normally secured with mutual TLS (mTLS): the server presents a certificate, and it also requires clients to present a certificate signed by a CA it trusts. When a client connects without a valid client certificate, or with one signed by an untrusted CA, the etcd server rejects the TLS handshake and the client sees remote error: tls: bad certificate. This is the access control working: an unauthenticated or wrongly-credentialed client must not read or write cluster state. It also fires when a legitimate operator points etcdctl at the wrong certs, forgets a flag, or uses an expired cert.

The literal client-side error from etcdctl:

{"level":"warn","msg":"retrying of unary invoker failed","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: remote error: tls: bad certificate\""}
Error: context deadline exceeded

On the etcd server, the rejected handshake is logged with the peer address:

{"level":"warn","msg":"rejected connection","remote-addr":"10.0.0.15:44992","server-name":"","error":"tls: client didn't provide a certificate"}
{"level":"warn","msg":"rejected connection","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority"}

The server log is the authoritative source: it says whether the client sent no certificate, a certificate from an unknown authority, or an expired one.

Symptoms

• etcdctl commands hang then fail with remote error: tls: bad certificate or context deadline exceeded.
• Kubernetes apiserver logs etcd connection failures; the cluster can’t read/write to etcd.
• The command works with one set of --cert/--key/--cacert flags but not another.
• Failures begin exactly when a certificate’s notAfter date passes.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 endpoint health

{"level":"warn",...,"error":"...remote error: tls: bad certificate"}
https://127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded

sudo journalctl -u etcd --since '5 min ago' | grep -i 'rejected connection\|certificate' | tail -3

... rejected connection ... error":"tls: client didn't provide a certificate"

Common Root Causes

1. No client certificate provided

etcdctl was invoked without --cert/--key, so the client presents nothing to a server that requires client auth (--client-cert-auth=true).

sudo journalctl -u etcd | grep "didn't provide a certificate" | tail -1

... "tls: client didn't provide a certificate"

2. Client cert signed by a CA the server doesn’t trust

The client cert is valid but issued by a different CA than the one in the server’s --trusted-ca-file, so verification fails as unknown authority.

sudo journalctl -u etcd | grep 'unknown authority' | tail -1

... x509: certificate signed by unknown authority

3. Expired client (or CA) certificate

etcd certs, especially in self-managed clusters, expire. An expired client cert is rejected even though everything else is correct.

openssl x509 -in /etc/etcd/pki/client.crt -noout -enddate

notAfter=Jul  8 12:00:00 2026 GMT

4. Wrong cert/key pair or mismatched key

--cert and --key don’t correspond to each other (mixed-up files), so the handshake fails.

openssl x509 -in client.crt -noout -modulus | openssl md5
openssl rsa  -in client.key -noout -modulus | openssl md5

Different digests mean the cert and key don’t match.

5. Missing Extended Key Usage (clientAuth)

The client cert lacks the TLS Web Client Authentication EKU, so a strict server refuses it for client auth.

openssl x509 -in client.crt -noout -ext extendedKeyUsage

X509v3 Extended Key Usage: TLS Web Server Authentication

(No TLS Web Client Authentication — invalid for client auth.)

6. Wrong CA passed to etcdctl (server-side verify)

--cacert points at a CA that doesn’t match the server’s cert, so the client can’t validate the server (a related but distinct handshake failure).

Diagnostic Workflow

Step 1: Read the server log to classify the failure

sudo journalctl -u etcd --since '10 min ago' | grep -i 'rejected connection\|certificate'

The message tells you which case you’re in: no cert, unknown authority, or expired. Fix the case the server names — don’t guess.

Step 2: Provide the full mTLS flag set

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/pki/ca.crt \
  --cert=/etc/etcd/pki/client.crt \
  --key=/etc/etcd/pki/client.key \
  endpoint health

Missing any of --cacert, --cert, --key is the most common self-inflicted cause.

Step 3: Verify the client cert is signed by the server’s trusted CA

# The CA etcd trusts for clients:
grep trusted-ca-file /etc/etcd/etcd.conf.yml
openssl verify -CAfile /etc/etcd/pki/ca.crt /etc/etcd/pki/client.crt

/etc/etcd/pki/client.crt: OK

OK confirms the CA chain; a failure here is your “unknown authority.”

Step 4: Check expiry on the cert and CA

for f in ca.crt client.crt; do
  echo "$f:"; openssl x509 -in /etc/etcd/pki/$f -noout -enddate
done
date -u

Compare notAfter to now; an expired cert must be reissued.

Step 5: Confirm the cert/key pair and EKU

openssl x509 -in client.crt -noout -modulus | openssl md5
openssl rsa  -in client.key -noout -modulus | openssl md5   # digests must match
openssl x509 -in client.crt -noout -ext extendedKeyUsage    # must include clientAuth

Example Root Cause Analysis

Overnight, a Kubernetes control plane loses etcd connectivity and etcdctl endpoint health fails with remote error: tls: bad certificate. The flags and files haven’t changed, which points at expiry. The server log is explicit:

sudo journalctl -u etcd --since '15 min ago' | grep -i certificate | tail -1

... rejected connection ... error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid"

Checking the client cert confirms it:

openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -enddate
date -u

notAfter=Jul  8 12:00:00 2026 GMT
Wed Jul  9 03:14:00 UTC 2026

The apiserver’s etcd client certificate expired at noon the previous day. mTLS then correctly rejected the now-invalid credential — the security control did exactly what it should; the failure is expired PKI, not a config error.

Fix: reissue the etcd client certificate from the cluster CA (here via kubeadm’s cert renewal), restart the consumer, and re-verify:

sudo kubeadm certs renew apiserver-etcd-client
openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -enddate
sudo systemctl restart kubelet    # restarts the static apiserver pod
ETCDCTL_API=3 etcdctl --cacert=... --cert=... --key=... endpoint health

notAfter=Jul  9 03:20:00 2027 GMT
https://127.0.0.1:2379 is healthy: successfully committed proposal

With a freshly issued, in-date certificate, the mTLS handshake succeeds and the control plane recovers — no TLS was disabled to get there.

Prevention Best Practices

• Monitor and alert on etcd (and all cluster PKI) certificate expiry well before notAfter; expiry is the most common and most avoidable cause of these outages.
• Automate certificate renewal (kubeadm certs renew, cert-manager, or your CA’s rotation) rather than hand-managing long-lived etcd certs.
• Keep --client-cert-auth=true and a correct --trusted-ca-file on etcd; never “fix” a handshake by disabling client-cert auth or TLS.
• Issue client certs with the clientAuth EKU from the same CA etcd trusts, and store cert/key pairs together so they can’t be mismatched.
• Wrap etcdctl flags in an alias or ETCDCTL_* environment variables so the full --cacert/--cert/--key set is always supplied consistently.
• Verify the chain with openssl verify -CAfile as part of any PKI change, before rolling it to production etcd nodes.

Quick Command Reference

# Classify the failure from the server side
sudo journalctl -u etcd --since '10 min ago' | grep -i 'rejected connection\|certificate'

# Full mTLS etcdctl call
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/pki/ca.crt --cert=/etc/etcd/pki/client.crt --key=/etc/etcd/pki/client.key \
  endpoint health

# Verify the client cert against etcd's trusted CA
openssl verify -CAfile /etc/etcd/pki/ca.crt /etc/etcd/pki/client.crt

# Check expiry
openssl x509 -in client.crt -noout -enddate; date -u

# Confirm cert/key match and clientAuth EKU
openssl x509 -in client.crt -noout -modulus | openssl md5
openssl rsa  -in client.key -noout -modulus | openssl md5
openssl x509 -in client.crt -noout -ext extendedKeyUsage

Conclusion

remote error: tls: bad certificate from etcd means the server rejected the client’s TLS handshake — mTLS access control refusing an unauthenticated or invalid client. Read the server log to classify it:

1. No client certificate provided — supply --cert/--key.
2. A cert signed by a CA etcd doesn’t trust — use one from its --trusted-ca-file.
3. An expired client or CA certificate — reissue it (the most common cause).
4. A mismatched cert/key pair.
5. A client cert missing the clientAuth EKU.
6. The wrong --cacert, breaking server verification.

Let the etcd log name the case, supply the full correct mTLS flag set, verify the chain and expiry with openssl, and reissue certificates as needed — restoring access without ever disabling etcd’s client-cert authentication.