🚨 OpenTelemetry Error Guide: 'rpc error: code = Unauthenticated' — Fix OTLP gRPC Auth

Overview

This error appears when an OTLP gRPC export reaches the backend but is rejected for missing or invalid credentials. TLS may be perfectly fine — the failure is at the application/auth layer, carried as a gRPC status code rather than an HTTP 401. It shows up in SDK stderr and in the Collector’s otlp exporter logs:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unauthenticated desc = authentication failed"}

You may see the credential named explicitly by the backend:

rpc error: code = Unauthenticated desc = authentication failed: invalid or missing bearer token

rpc error: code = Unauthenticated desc = authentication failed is gRPC status code 16 — the backend received the request but the bearer token or API key was absent, expired, or wrong. It is the gRPC analogue of an HTTP 401 Unauthorized.

Symptoms

• Every gRPC export is rejected with code = Unauthenticated, deterministically, regardless of payload.
• The TLS handshake succeeds — this is not a certificate error — but auth fails afterward.
• The same credential works over HTTP OTLP but fails over gRPC (headers set on the wrong exporter).
• The backend or vendor Collector requires an API key / bearer token that the exporter never sends.
• Rotating in a fresh, valid token immediately clears the error.
• Logs may name the missing header (authorization, x-api-key) explicitly.

Common Root Causes

• Missing auth header on the gRPC exporter — headers (or the bearertokenauth extension) was set on the HTTP exporter but not the gRPC one.
• Expired or rotated token — the API key was rotated at the backend but the exporter still carries the old value.
• Wrong header name — the backend expects authorization: Bearer <token> but the config sends x-api-key, or vice versa.
• Token not interpolated — a literal ${API_KEY} was shipped because the environment variable was never set or expanded.
• Auth extension not wired into the pipeline — a bearertokenauth extension exists but is not referenced by the exporter’s auth.authenticator.
• Scoped/tenant mismatch — a valid token for the wrong project or tenant is rejected as unauthenticated.

Diagnostic Workflow

Confirm the credential the exporter is actually sending, and that it is populated rather than a literal placeholder:

# Is the token present and non-empty? (do not print secrets in shared logs)
test -n "$OTLP_BEARER_TOKEN" && echo "token set (len ${#OTLP_BEARER_TOKEN})" || echo "TOKEN MISSING"
env | grep -E 'OTEL_EXPORTER_OTLP_HEADERS|OTLP_BEARER_TOKEN'

The cleanest way to attach a gRPC bearer token in the Collector is the bearertokenauth extension, referenced from the exporter’s auth block:

extensions:
  bearertokenauth:
    scheme: Bearer
    token: ${env:OTLP_BEARER_TOKEN}    # injected from env; never hard-code

exporters:
  otlp:
    endpoint: otlp.vendor.example.com:4317
    tls:
      insecure: false
      ca_file: /etc/otelcol/certs/vendor-ca.crt
    auth:
      authenticator: bearertokenauth

service:
  extensions: [bearertokenauth]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

If the backend expects a raw API-key header instead of a bearer scheme, set it directly on the exporter:

exporters:
  otlp:
    endpoint: otlp.vendor.example.com:4317
    headers:
      x-api-key: ${env:OTLP_API_KEY}   # exact header name the backend documents
    tls:
      insecure: false

For an SDK exporter, the equivalent header goes in the OTLP headers env var (keep the token redacted in any shared output):

export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${OTLP_BEARER_TOKEN}"

Validate and watch for the auth failure clearing:

otelcol-contrib validate --config /etc/otelcol/config.yaml
journalctl -u otelcol-contrib --since '10 min ago' | grep -i 'unauthenticated\|authentication failed'

Example Root Cause Analysis

A team migrated their export from OTLP/HTTP to OTLP/gRPC for lower overhead. The old HTTP exporter had carried the vendor API key in a headers block, but when they added the new otlp (gRPC) exporter they forgot to move the credential. TLS negotiated fine, yet every export failed with rpc error: code = Unauthenticated desc = authentication failed, and the vendor UI showed zero incoming spans.

The fix had two parts. First, they added a bearertokenauth extension sourcing the key from ${env:OTLP_BEARER_TOKEN} and referenced it via auth.authenticator on the gRPC exporter, so the authorization: Bearer header was attached to every gRPC call. Second, they moved the token out of the config file entirely into the systemd unit’s environment and rotated the previously committed key, closing the leak. After a reload the Unauthenticated errors stopped and spans appeared in the vendor within seconds.

Prevention Best Practices

• Attach credentials with the bearertokenauth extension (or explicit headers) on every exporter, and re-check after any protocol change.
• Source tokens from the environment (${env:...}) or a secrets manager — never commit them to the Collector config.
• Rotate keys on a schedule and roll the new value to exporters before the old one is revoked.
• Match the exact header name and scheme the backend documents (authorization: Bearer vs x-api-key).
• Alert on otelcol_exporter_send_failed_spans and on Unauthenticated log lines so a bad or expired token pages quickly.
• Keep HTTP and gRPC exporter auth in sync when you run both, so a migration never silently drops credentials.

Quick Command Reference

# Confirm the token is set without printing it
test -n "$OTLP_BEARER_TOKEN" && echo "token present" || echo "TOKEN MISSING"

# Inspect OTLP header env (redact before sharing)
env | grep OTEL_EXPORTER_OTLP_HEADERS

# Validate config after adding the auth extension/headers
otelcol-contrib validate --config /etc/otelcol/config.yaml

# Watch the Unauthenticated error clear
journalctl -u otelcol-contrib -f | grep -i 'unauthenticated\|authentication failed'

Conclusion

rpc error: code = Unauthenticated desc = authentication failed is gRPC status 16 — the request arrived but the credential was missing, expired, or wrong. It is distinct from an HTTP 401 and from any TLS error: the handshake succeeds, then auth fails. Attach a valid bearer token with the bearertokenauth extension or a documented API-key header, source it from the environment rather than the config file, and keep it rotated. Get the header name, scheme, and value right on the gRPC exporter and telemetry flows again.

Related

• OpenTelemetry Error Guide: ‘401 Unauthorized’ exporting traces
• OpenTelemetry Error Guide: ‘x509: certificate signed by unknown authority’
• OpenTelemetry Error Guide: ‘rpc error: code = Unimplemented’