🚨 OpenTelemetry Error Guide: 'first record does not look like a TLS handshake' — Fix Plaintext vs TLS OTLP

Overview

This error is the mirror image of a plaintext/TLS mix-up, seen from the server’s side. The Collector’s OTLP receiver is configured with TLS, but a client connected in plaintext and sent unencrypted bytes where a TLS ClientHello was expected. The receiver logs:

error	otelarrow/internal.go	error reading from client	{"kind": "receiver", "name": "otlp", "transport": "grpc", "error": "tls: first record does not look like a TLS handshake"}

The same message can appear on the client when a plaintext SDK exporter connects to a TLS listener and the raw stream is rejected:

rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

tls: first record does not look like a TLS handshake means one side expects TLS and the other spoke cleartext — typically a gRPC exporter left insecure while the receiver has a tls: block (or vice versa).

Symptoms

• The Collector receiver logs the error repeatedly while no telemetry is accepted.
• The failure is deterministic and starts exactly when TLS was enabled on one side only.
• A gRPC exporter has tls.insecure: true (or the SDK OTEL_EXPORTER_OTLP_INSECURE=true) but the receiver requires TLS.
• curl http://endpoint or grpcurl -plaintext fails against a TLS-enabled port.
• Enabling TLS on the client, or disabling it on the receiver, immediately clears the error.
• The reverse case: a plaintext receiver logs nothing but the client fails with server gave HTTP response to HTTPS client.

Common Root Causes

• gRPC exporter left insecure — tls.insecure: true on the exporter while the receiver has a tls: block, so plaintext bytes hit a TLS listener.
• Receiver TLS added on one side only — someone enabled tls: on the receiver without updating the agents that still export in plaintext.
• Wrong port — the client aims at the TLS port but sends cleartext, or at the plaintext port assuming TLS.
• OTEL_EXPORTER_OTLP_INSECURE=true set in the SDK against a secure Collector endpoint.
• A plaintext health check or scanner hitting the TLS OTLP port and logging noise that looks like the real client failing.
• Protocol confusion — an http/protobuf client posting to a gRPC TLS port, so nothing about the stream matches a handshake.

Diagnostic Workflow

Determine which side is plaintext. Inspect the exporter’s TLS mode first:

echo "$OTEL_EXPORTER_OTLP_ENDPOINT"    # host:port or scheme
echo "$OTEL_EXPORTER_OTLP_INSECURE"    # true = plaintext gRPC; a red flag vs a TLS receiver
env | grep OTEL_EXPORTER_OTLP

Probe the port. If a plaintext probe is rejected, the port is TLS and the client must use TLS too:

# Against a TLS gRPC port, plaintext is refused mid-stream:
grpcurl -plaintext otel-gateway.example.com:4317 list
# TLS probe should succeed and list services:
grpcurl otel-gateway.example.com:4317 list

If the receiver should be TLS, fix the client: turn off insecure mode and trust the server CA:

exporters:
  otlp:
    endpoint: otel-gateway.example.com:4317
    tls:
      insecure: false                 # attempt a real TLS handshake
      ca_file: /etc/otelcol/certs/server-ca.crt

The receiver keeps its TLS block, which is what triggered the message when a plaintext client connected:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/otelcol/certs/server.crt
          key_file: /etc/otelcol/certs/server.key

If instead the receiver is meant to be plaintext (dev/local), remove its tls: block or move TLS to a dedicated port, and set tls.insecure: true on the exporter to match. Validate and watch the receiver:

otelcol-contrib validate --config /etc/otelcol/config.yaml
journalctl -u otelcol-contrib --since '10 min ago' | grep -i 'handshake\|tls\|reading from client'

Example Root Cause Analysis

An SRE enabled TLS on the gateway Collector’s OTLP gRPC receiver during a security pass, adding cert_file and key_file. The DaemonSet agents, however, still exported with tls.insecure: true. The gateway immediately began logging tls: first record does not look like a TLS handshake on every connection, because the agents were sending raw gRPC frames where the server expected a ClientHello, and no spans were accepted.

The fix was to bring the clients up to TLS rather than roll the server back. Each agent’s otlp exporter had insecure flipped to false and was given ca_file pointing at the internal CA that signed the gateway certificate. A quick grpcurl otel-gateway.example.com:4317 list (without -plaintext) confirmed the TLS listener answered. After the rolling restart, the handshake errors stopped and the receiver began accepting encrypted OTLP traffic with no gaps.

Prevention Best Practices

• Change TLS on the receiver and the exporters together; never enable server TLS while clients still export in plaintext.
• Treat tls.insecure: true and OTEL_EXPORTER_OTLP_INSECURE=true as dev-only and audit for them before enabling receiver TLS.
• Use separate ports for plaintext and TLS listeners so a cleartext client can never accidentally hit a TLS port.
• Probe with grpcurl (TLS) and grpcurl -plaintext in CI to assert the port behaves as intended.
• Keep the server CA bundle distributed to every exporter so switching insecure off is a one-line change.
• Alert on receiver-side reading from client / handshake log lines to catch a one-sided TLS rollout instantly.

Quick Command Reference

# Show whether the client is in insecure (plaintext) mode
env | grep OTEL_EXPORTER_OTLP

# Plaintext probe fails against a TLS port; TLS probe succeeds
grpcurl -plaintext otel-gateway.example.com:4317 list
grpcurl otel-gateway.example.com:4317 list

# Validate config after aligning tls/insecure
otelcol-contrib validate --config /etc/otelcol/config.yaml

# Watch the receiver reject plaintext connections
journalctl -u otelcol-contrib -f | grep -i 'handshake\|reading from client'

Conclusion

tls: first record does not look like a TLS handshake is a plaintext client meeting a TLS server. Decide which side is authoritative: if the receiver should be secure, set tls.insecure: false and a ca_file on every exporter; if it should be plaintext, remove the receiver’s tls: block or move TLS to its own port and set the exporter to insecure. The rule that prevents recurrence is simple — flip TLS on the receiver and its clients in the same change, and keep secure and plaintext listeners on separate ports.

Related

• OpenTelemetry Error Guide: ‘x509: certificate signed by unknown authority’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘connection reset by peer’ on OTLP export