🚨 OpenTelemetry Error Guide: 'rpc error: code = PermissionDenied' — Fix OTLP Authorization

Overview

This error appears when the backend successfully authenticates the request but decides the caller is not authorized for the target dataset, project, or signal type. gRPC returns PermissionDenied, and the Collector’s exporter helper logs it:

2026-07-12T11:07:44.219Z	error	exporterhelper/queue_sender.go:174	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "rpc error: code = PermissionDenied desc = permission denied", "dropped_items": 1024}

A backend often adds detail in the desc:

rpc error: code = PermissionDenied desc = permission denied: api key lacks scope "metrics:write" for project "prod-eu"

PermissionDenied is distinct from Unauthenticated (code 16): the credential is valid, but it does not grant access to what you asked for. This is a permanent, non-retryable condition — retrying the same key against the same resource will fail identically, so the batch is dropped.

Symptoms

• Export fails with code = PermissionDenied immediately, with no retry/backoff delay.
• One signal (e.g. metrics) is rejected while another (traces) using a different key succeeds.
• The error started after a key rotation, an org/project migration, or a scope/role change.
• desc mentions a missing scope, wrong project, or a dataset the key can’t write to.
• otelcol_exporter_send_failed_metrics climbs while the queue stays low (nothing retried).

Common Root Causes

• Wrong scope on the API key — the key authenticates but lacks the traces:write / metrics:write scope the endpoint requires.
• Key belongs to a different project/org — a valid key from staging is used against a prod dataset it has no rights to.
• Wrong dataset/tenant header — the backend routes by a header (e.g. x-dataset) that names a resource the key can’t access.
• Read-only or expired-scope credential — the key was provisioned for querying, not ingestion.
• RBAC/role misconfiguration — the service account maps to a role without ingest permission for that signal.
• Confusing PermissionDenied with Unauthenticated — treating a scope problem as a “bad token” and rotating the key without fixing scope.

Diagnostic Workflow

First confirm this is authorization, not authentication — check the exact code and desc:

journalctl -u otelcol-contrib --since '10 min ago' | grep -i 'PermissionDenied\|permission denied\|scope'

Inspect how the exporter presents credentials. Keep the key out of the file by sourcing it from the environment:

exporters:
  otlp:
    endpoint: ingest.example.com:4317
    tls:
      insecure: false
    headers:
      api-key: ${API_KEY}
      x-dataset: prod-eu        # tenant/dataset the key must be scoped for

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

Verify the key’s scope out-of-band by sending a minimal request to the backend’s ingest path with the same credential and dataset header:

# Confirm the credential resolves and check the returned status/detail
grpcurl -H "api-key: ${API_KEY}" -H "x-dataset: prod-eu" \
  ingest.example.com:443 list

# HTTP/OTLP path: a 403 body usually names the missing scope
curl -si https://ingest.example.com:4318/v1/metrics \
  -H "api-key: ${API_KEY}" -H "x-dataset: prod-eu" \
  -H "Content-Type: application/x-protobuf" --data-binary @/dev/null | head -20

If the backend uses a Collector-side auth extension, confirm it is wired to the exporter and reading the right secret:

otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml
env | grep -E 'API_KEY|OTLP'   # confirm the secret is present (value redacted in logs)

Example Root Cause Analysis

A platform team migrated ingestion from a single shared dataset to per-environment datasets (prod-eu, prod-us, staging). They rolled out a new x-dataset: prod-eu header to the EU gateway Collector but reused the staging API key that was still sitting in the EU secret. The key authenticated fine, so no Unauthenticated error appeared — instead every metrics export returned rpc error: code = PermissionDenied desc = permission denied because the staging key had no metrics:write scope on prod-eu. Since the error was permanent, 100% of EU metrics were silently dropped.

The fix had two parts. First, a correctly scoped prod-eu ingest key was minted in the backend and written to the EU Collector’s secret, replacing the staging key. Second, the team added a startup smoke check that posts an empty OTLP metrics request with the live credential and dataset header, failing the deploy if the backend answers PermissionDenied. After redeploy, metrics flowed to prod-eu and the smoke check now catches scope mismatches before they reach production.

Prevention Best Practices

• Treat PermissionDenied and Unauthenticated as different problems: one is scope/authorization, the other is a bad or missing credential — don’t fix one by rotating for the other.
• Provision distinct ingest keys per project/dataset with least-privilege *:write scopes, and label secrets so the right key lands in the right Collector.
• Keep credentials in environment variables or a secret store referenced as ${API_KEY}; never commit them to config.
• Add a deploy-time smoke test that sends an empty OTLP request with the real credential and dataset header and fails on PermissionDenied.
• Audit dataset/tenant headers (e.g. x-dataset) alongside the key so both agree on the target resource.
• Alert on otelcol_exporter_send_failed_* with a low queue size — the signature of permanent authorization drops.

Quick Command Reference

# Confirm it's authorization (PermissionDenied), not auth (Unauthenticated)
journalctl -u otelcol-contrib -f | grep -i 'PermissionDenied\|Unauthenticated'

# Test the live key + dataset header against the ingest endpoint
curl -si https://ingest.example.com:4318/v1/metrics \
  -H "api-key: ${API_KEY}" -H "x-dataset: prod-eu" \
  -H "Content-Type: application/x-protobuf" --data-binary @/dev/null | head

# Validate the exporter/auth config
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

# Watch permanent-drop counters climb with a low queue
curl -s http://localhost:8888/metrics | grep -E 'exporter_send_failed|exporter_queue_size'

Conclusion

rpc error: code = PermissionDenied means your credential is genuine but not entitled to the dataset, project, or signal you targeted — a scope problem, not a token problem. Read the desc for the missing scope, confirm the API key and any dataset/tenant header actually match the resource, and provision a least-privilege ingest key that does. Because the error is permanent and drops data instantly, a deploy-time smoke check against the live endpoint is the surest way to catch a mis-scoped key before it silently blackholes a whole signal.

Related

• OpenTelemetry Error Guide: ‘401 Unauthorized’ exporting traces
• OpenTelemetry Error Guide: ‘rpc error: code = Unimplemented’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector