🚨 OpenTelemetry Error Guide: '401 Unauthorized' exporting traces — Fix OTLP Auth Headers

Overview

This error appears when an OpenTelemetry exporter reaches an authenticated OTLP backend but the request lacks valid credentials. The backend rejects it and the exporter marks the failure permanent:

2026-07-09T15:26:33.912Z	error	exporterhelper/common.go:296	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "error exporting items, request to https://otlp.vendor.com/v1/traces responded with HTTP Status Code 401, Message=Unauthorized: invalid API key"}

The gRPC transport variant carries a status code instead:

rpc error: code = Unauthenticated desc = missing or invalid authentication token

401 Unauthorized (HTTP) / Unauthenticated (gRPC) means the credential is missing, malformed, or expired. A related 403 Forbidden / PermissionDenied means the credential is valid but lacks permission for that endpoint or dataset.

Symptoms

• Every export to the authenticated backend fails with 401/403 and is dropped (non-retryable).
• Local/unauthenticated pipelines work, but the vendor backend rejects everything.
• The error message names an invalid, missing, or expired API key/token.
• Started failing after a key rotation, secret change, or endpoint migration.
• gRPC exports show code = Unauthenticated or code = PermissionDenied.

Common Root Causes

• Missing auth header — OTEL_EXPORTER_OTLP_HEADERS not set, so no API key is sent.
• Wrong header name/format — the backend expects x-api-key or Authorization: Bearer ... and got neither.
• Expired or rotated key — the secret in the pod/exporter is stale.
• Wrong endpoint/tenant — key is valid but for a different region, project, or dataset (403).
• Secret not mounted — the env var references an unset secret, so the header is empty.
• TLS terminated wrong — a proxy strips the header, or insecure skips required mTLS.

Diagnostic Workflow

For SDK exporters, pass the credential via headers. The value must match exactly what the backend expects:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.vendor.com"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${OTLP_API_TOKEN}"
# some backends use a custom header instead:
# export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=${OTLP_API_KEY}"

In the Collector, set the header on the exporter (never hard-code the secret; expand from env):

exporters:
  otlphttp:
    endpoint: https://otlp.vendor.com
    headers:
      authorization: "Bearer ${env:OTLP_API_TOKEN}"
    tls:
      insecure: false

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

For per-request bearer tokens across a pipeline, use the bearertokenauth extension:

extensions:
  bearertokenauth:
    token: "${env:OTLP_API_TOKEN}"

exporters:
  otlp:
    endpoint: otlp.vendor.com:4317
    auth:
      authenticator: bearertokenauth

service:
  extensions: [bearertokenauth]

Confirm the credential is present in the runtime and test the endpoint directly:

tr '\0' '\n' < /proc/$(pgrep -f otelcol)/environ | grep OTLP_API_TOKEN
curl -v -X POST "https://otlp.vendor.com/v1/traces" \
  -H "Authorization: Bearer ${OTLP_API_TOKEN}" \
  -H "Content-Type: application/x-protobuf" --data-binary @/dev/null

Example Root Cause Analysis

A vendor rotated API keys and the new key was written to a Kubernetes Secret, but the Collector Deployment still referenced the old Secret key name in its env. ${env:OTLP_API_TOKEN} expanded to an empty string, so the otlphttp exporter sent authorization: Bearer with no token. The backend returned HTTP Status Code 401, Unauthorized: invalid API key, and because auth failures are permanent, every span was dropped immediately.

Diagnosis confirmed the empty token by dumping the process environment and seeing OTLP_API_TOKEN blank. The fix updated the Deployment’s secretKeyRef to the rotated Secret and restarted the Collector; the environment then carried the real token and curl with the same Authorization header returned 200. To prevent recurrence, the team moved to the bearertokenauth extension reading a mounted file and added an alert on otelcol_exporter_send_failed_spans so an auth break pages instead of silently dropping data.

Prevention Best Practices

• Inject credentials from a secret manager or Kubernetes Secret, never hard-coded in the config.
• Use ${env:VAR} expansion and verify the variable is non-empty at startup.
• Match the exact header the backend documents (Authorization: Bearer vs x-api-key).
• Automate key rotation so exporters pick up new secrets without a manual config edit.
• Keep TLS enabled (insecure: false) to authenticated endpoints so headers are not stripped in transit.
• Alert on otelcol_exporter_send_failed_* and on 401/403 responses; auth failures are permanent drops.

Quick Command Reference

# Confirm the token is present and non-empty
tr '\0' '\n' < /proc/$(pgrep -f otelcol)/environ | grep OTLP_API_TOKEN

# Test the authenticated endpoint directly
curl -v -X POST "https://otlp.vendor.com/v1/traces" \
  -H "Authorization: Bearer ${OTLP_API_TOKEN}" \
  -H "Content-Type: application/x-protobuf" --data-binary @/dev/null

# SDK header configuration
env | grep OTEL_EXPORTER_OTLP_HEADERS

# Watch for auth failures in the Collector
journalctl -u otelcol-contrib | grep -iE '401|403|unauthenticated|permissiondenied'

Conclusion

401 Unauthorized / Unauthenticated when exporting traces is an authentication problem, not a network one: the backend was reached but rejected the credential. Send the exact header the backend expects (OTEL_EXPORTER_OTLP_HEADERS or a Collector bearertokenauth/headers entry), source the secret from a manager with ${env:...} expansion, and verify it is non-empty at runtime. Because auth failures are permanent drops, alert on them — a rotated key should page you, not quietly erase your telemetry.