🚨 OpenTelemetry Error Guide: 'remote error: tls: handshake failure' — Fix OTLP TLS Negotiation

Overview

This error appears when an OTLP client and the Collector open a TCP connection but cannot agree on TLS parameters, so the peer aborts the handshake. It shows up in SDK stderr and in the Collector’s otlp exporter logs:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: authentication handshake failed: remote error: tls: handshake failure\""}

The HTTP/OTLP variant reads:

traces export: Post "https://otel-gateway.example.com:4318/v1/traces": remote error: tls: handshake failure

remote error: tls: handshake failure means the server sent a TLS alert rejecting the handshake — usually a TLS version or cipher-suite mismatch, or a server that requires a client certificate (mTLS) that the exporter never presented.

Symptoms

• Every export fails immediately with tls: handshake failure; there is no partial success.
• The failure is deterministic, not intermittent — it never connects, unlike a reset that comes and goes.
• Plain tcp connectivity to the port succeeds, but TLS negotiation fails.
• The Collector receiver logs show the client disconnecting during the handshake, before any OTLP request.
• Adding a client cert and key (or lowering a TLS floor) makes the error disappear.
• openssl s_client against the endpoint also fails or reports “no shared cipher”.

Common Root Causes

• mTLS required but no client cert sent — the receiver sets client_ca_file and rejects any exporter that does not present cert_file/key_file.
• TLS version mismatch — the receiver enforces min_version: "1.3" while the client only offers TLS 1.2 (or an old client caps at 1.1).
• No shared cipher suite — a hardened cipher_suites list on the server excludes everything the client offers.
• Wrong protocol on the port — the exporter speaks TLS to a gRPC/HTTP plaintext listener, or vice versa, and the peer aborts.
• Corrupt or truncated server cert chain — a malformed cert_file makes the server abort mid-handshake.
• SNI / virtual-host mismatch at a proxy — an L7 proxy terminating TLS rejects the ClientHello because the SNI does not match any configured cert.

Diagnostic Workflow

First confirm what the exporter is actually connecting to and whether it presents a client certificate:

env | grep OTEL_EXPORTER_OTLP
echo "$OTEL_EXPORTER_OTLP_CERTIFICATE"          # server CA the client trusts
echo "$OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE"   # client cert for mTLS
echo "$OTEL_EXPORTER_OTLP_CLIENT_KEY"           # client key for mTLS

Probe the endpoint directly to see the negotiated version and any client-cert request. If the server asks for a certificate, openssl prints an “Acceptable client certificate CA names” list:

# Inspect the TLS negotiation and whether mTLS is demanded
openssl s_client -connect otel-gateway.example.com:4317 -servername otel-gateway.example.com </dev/null

# Force a version to prove a floor/ceiling mismatch
openssl s_client -connect otel-gateway.example.com:4317 -tls1_2 </dev/null

On the receiving Collector, configure TLS explicitly. For mTLS, the receiver names the CA that signs client certs; for a version floor, keep it compatible with your fleet:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/otelcol/certs/server.crt
          key_file: /etc/otelcol/certs/server.key
          client_ca_file: /etc/otelcol/certs/client-ca.crt   # requires mTLS
          min_version: "1.2"
          max_version: "1.3"

On the exporter, present the matching client certificate and trust the server CA:

exporters:
  otlp:
    endpoint: otel-gateway.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/certs/server-ca.crt
      cert_file: /etc/otelcol/certs/client.crt   # required when server sets client_ca_file
      key_file: /etc/otelcol/certs/client.key
      min_version: "1.2"

Validate the config and watch the receiver for handshake rejections:

otelcol-contrib validate --config /etc/otelcol/config.yaml
journalctl -u otelcol-contrib --since '10 min ago' | grep -i 'handshake\|tls\|certificate'

Example Root Cause Analysis

A team enabled mutual TLS on their gateway Collector by adding client_ca_file to the OTLP gRPC receiver, but only updated the server side. Every agent immediately began logging remote error: tls: handshake failure, and traces stopped arriving. openssl s_client against the gateway printed an “Acceptable client certificate CA names” block, proving the server was demanding a client cert the agents never sent.

The fix had two parts. First, each agent’s otlp exporter was given cert_file and key_file signed by the same internal CA listed in the gateway’s client_ca_file, so the handshake could complete. Second, the CA bundle was distributed via the existing secret-management path and the server’s min_version was pinned to "1.2" to match older agents still in the fleet. After rollout the handshake error cleared and mTLS was enforced end to end with zero dropped spans.

Prevention Best Practices

• When enabling mTLS, roll out client certs to exporters before setting client_ca_file on the receiver, not after.
• Pin min_version/max_version explicitly on both ends so a library upgrade can’t silently drop a shared version.
• Keep cipher_suites lists in sync between client and server when you harden them; test with openssl s_client first.
• Automate certificate distribution and renewal so client certs never drift out of sync with the receiver’s CA.
• Alert on the Collector’s otelcol_receiver_refused_spans and on repeated handshake log lines to catch fleet-wide breakage fast.
• Never mix a plaintext listener and a TLS exporter on the same port — keep secure and insecure endpoints clearly separated.

Quick Command Reference

# Show the exporter's TLS-related environment
env | grep OTEL_EXPORTER_OTLP

# Negotiate TLS and reveal any client-cert demand
openssl s_client -connect otel-gateway.example.com:4317 -servername otel-gateway.example.com </dev/null

# Prove a TLS version floor/ceiling mismatch
openssl s_client -connect otel-gateway.example.com:4317 -tls1_2 </dev/null

# Validate Collector config after editing tls blocks
otelcol-contrib validate --config /etc/otelcol/config.yaml

# Watch the receiver reject handshakes live
journalctl -u otelcol-contrib -f | grep -i 'handshake\|tls'

Conclusion

remote error: tls: handshake failure is the server aborting TLS negotiation before any telemetry flows. The cause is almost always a mismatch you can enumerate: a required client cert the exporter never sent, an incompatible min_version/max_version, or a cipher_suites list with no overlap. Reproduce the handshake with openssl s_client, align the tls: blocks on both the receiver and the exporter, and roll out mTLS certificates before you enforce them — then the handshake completes and OTLP data resumes.

Related

• OpenTelemetry Error Guide: ‘x509: certificate signed by unknown authority’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘connection reset by peer’ on OTLP export