🚨 OpenTelemetry Error Guide: 'x509: certificate signed by unknown authority' — Fix OTLP TLS Trust

Overview

When the OTLP exporter connects to a Collector or backend over TLS, Go verifies the server certificate against a trusted CA set. If the presented certificate chains to a CA the client does not trust, the export fails with:

Exporting failed. Will retry the request after interval.
error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

A closely related variant appears when the certificate is trusted but the hostname does not match its SAN list:

tls: failed to verify certificate: x509: certificate is valid for otel-gateway, not otel-collector.observability.svc

Until trust is established, every export is rejected at the handshake and telemetry never leaves the process.

Symptoms

• No traces/metrics reach the backend; the exporter retries continuously.
• Logs show x509: certificate signed by unknown authority or certificate is valid for X, not Y.
• Plaintext export to the same host works, but TLS fails.
• The error started after enabling TLS/mTLS, rotating a CA, or switching to a self-signed/internal CA.
• curl from the pod also fails verification against the same endpoint.

Common Root Causes

• Missing CA in the client trust store — the endpoint uses an internal/self-signed CA that isn’t in the exporter’s ca_file or the image’s system bundle.
• Wrong or stale CA bundle — a rotated CA where the client still has the old root.
• Hostname/SAN mismatch — connecting via a name (or IP) not listed in the certificate’s Subject Alternative Names.
• mTLS misconfig — the server requires a client cert the exporter isn’t presenting, surfacing as a handshake failure.
• cert-manager not mounted — the Secret with ca.crt isn’t mounted into the Collector pod, or the path is wrong.

Diagnostic Workflow

Inspect the exporter TLS block; the client must point at the CA that signed the server certificate:

exporters:
  otlp:
    endpoint: otel-gateway.observability.svc:4317
    tls:
      insecure: false
      ca_file: /etc/otel/certs/ca.crt
      # mTLS: present a client cert if the server requires one
      cert_file: /etc/otel/certs/tls.crt
      key_file: /etc/otel/certs/tls.key

Confirm the CA file is actually present in the container and matches the server’s issuer:

kubectl exec deploy/otel-agent -- ls -l /etc/otel/certs/
openssl s_client -connect otel-gateway.observability.svc:4317 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject -ext subjectAltName

Compare the SAN list from that output to the endpoint you configured — the hostname must appear in the SANs. For SDK exporters, point at the CA via environment variables:

export OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/otel/certs/ca.crt
export OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE=/etc/otel/certs/tls.crt
export OTEL_EXPORTER_OTLP_CLIENT_KEY=/etc/otel/certs/tls.key

Watch the Collector logs to confirm the handshake succeeds after the fix:

journalctl -u otelcol -f | grep -i 'x509\|handshake\|tls'

Example Root Cause Analysis

After migrating an OTLP gateway behind cert-manager, every agent began logging x509: certificate signed by unknown authority. The gateway certificate was issued by an internal ClusterIssuer, but the agents’ otlp exporter had no ca_file set, so Go fell back to the system trust store, which had no knowledge of the internal CA.

The fix was to mount the CA Secret (ca.crt) into the agent pods and set ca_file: /etc/otel/certs/ca.crt on the exporter. A second wave of errors — certificate is valid for otel-gateway, not 10.0.4.12 — appeared because one agent targeted the gateway by pod IP; switching the endpoint to the Service DNS name that was present in the cert’s SANs resolved it. After both changes, handshakes succeeded and the retry backoff cleared.

Prevention Best Practices

• Always set an explicit ca_file (or SDK OTEL_EXPORTER_OTLP_CERTIFICATE) when using internal/self-signed CAs instead of relying on the system bundle.
• Address endpoints by a DNS name listed in the certificate’s SANs; never connect by raw IP unless the IP is in the SANs.
• Automate CA distribution and rotation with cert-manager and mount the CA Secret into every client; roll new roots before revoking old ones.
• Keep server and client cert lifetimes and rotation in sync so a rotation never leaves clients trusting a retired root.
• Add a synthetic export check over TLS in CI/staging so trust breaks are caught before production.

Quick Command Reference

# Show the server cert issuer and SANs
openssl s_client -connect HOST:4317 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -noout -issuer -subject -ext subjectAltName

# Verify the CA file is mounted in the pod
kubectl exec deploy/otel-agent -- ls -l /etc/otel/certs/

# Point the SDK exporter at the CA
export OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/otel/certs/ca.crt

# Watch handshakes
journalctl -u otelcol -f | grep -i 'x509\|handshake'

Conclusion

x509: certificate signed by unknown authority means the OTLP client does not trust the CA that signed the server’s certificate, and the related SAN-mismatch variant means the hostname isn’t covered by the cert. Fix it by supplying the correct ca_file/CA bundle, presenting a client cert for mTLS, and addressing endpoints by a SAN-listed DNS name. Automating CA distribution and rotation with cert-manager keeps TLS trust intact through every rollout.