🚨 OpenTelemetry Error Guide: 'Permanent error: rpc error: code = InvalidArgument' — Fix Rejected OTLP Data

Overview

This error appears when the backend accepts the connection and reads the request, but rejects the OTLP payload itself as invalid. gRPC classifies InvalidArgument as a client-side fault, so the Collector’s exporter helper marks it permanent — it is not retried, and the batch is dropped immediately:

2026-07-12T09:41:22.503Z	error	exporterhelper/queue_sender.go:174	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "Permanent error: rpc error: code = InvalidArgument desc = invalid ... : malformed data", "dropped_items": 512}

A common variant names the offending field the backend could not decode:

Permanent error: rpc error: code = InvalidArgument desc = invalid resource attribute "service.name": expected string value

InvalidArgument means the server understood the request but the data violated a schema or protocol rule — a bad resource attribute, an unsupported field, or a proto/version mismatch. Because retrying identical data would fail identically, the exporter drops it rather than looping.

Symptoms

• Traces, metrics, or logs never arrive for one specific service or pipeline while others are fine.
• Collector logs Permanent error with code = InvalidArgument and a rising dropped_items count.
• Errors appear immediately on export, not after a retry/backoff delay.
• otelcol_exporter_send_failed_spans climbs while otelcol_exporter_queue_size stays low (nothing is being retried).
• The failure started right after a schema change, an SDK upgrade, or a new processor was added.

Common Root Causes

• Invalid resource or span attributes — an attribute has the wrong type (e.g. an int where the backend expects a string) or a reserved key is overwritten by a transform/attributes processor.
• Unsupported or unknown field — the payload carries a proto field the backend’s OTLP version does not accept.
• Protocol/encoding mismatch — sending OTLP/JSON to a proto-only endpoint, or gRPC data to an HTTP path, so the body cannot be decoded.
• Missing required field — the backend mandates service.name or a valid timestamp and rejects records that lack it.
• Oversized or malformed strings — non-UTF-8 attribute values or values exceeding the backend’s length limits.
• Vendor-specific constraints — a hosted backend enforces stricter validation (attribute count, metric temporality) than the OTLP spec baseline.

Diagnostic Workflow

First identify which pipeline and exporter is dropping data. Turn on verbose exporter logging so the backend’s desc string is fully captured:

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Read the log line and note the exact field the backend names in desc:

journalctl -u otelcol-contrib --since '10 min ago' | grep -i 'InvalidArgument\|Permanent error\|malformed'

Route a copy of the traffic to a debug exporter to see the actual payload the backend is rejecting, without touching the real pipeline:

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: ingest.example.com:4317
    tls:
      insecure: false
    headers:
      api-key: ${API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlp]

Confirm the protocol matches the endpoint you are targeting. A gRPC exporter must point at the gRPC port (4317), and OTLP/HTTP at 4318:

# gRPC endpoint should answer a reflection/list call
grpcurl -plaintext ingest.example.com:4317 list
# HTTP endpoint expects proto on /v1/traces, not the gRPC port
curl -v -m 5 https://ingest.example.com:4318/v1/traces

If a processor is mutating attributes, validate the config and inspect the transform/attributes stages that could produce a bad type:

otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

Example Root Cause Analysis

A team added a transform processor to normalize environment labels and shipped a statement that coerced service.name from a resource attribute into an integer index for internal routing. Their vendor backend requires service.name to be a string, so every export came back as Permanent error: rpc error: code = InvalidArgument desc = invalid resource attribute "service.name": expected string value. Because the error was permanent, none of it was retried — the service simply vanished from the backend while the Collector quietly incremented dropped_items.

The fix had two parts. First, the offending transform statement was corrected to write the numeric routing hint to a new attribute (app.route_id) and leave service.name as the original string. Second, a debug exporter was wired into the pipeline in staging so the exact rejected payload was visible before promotion. After redeploy, the backend accepted every batch and the service reappeared with zero permanent errors.

Prevention Best Practices

• Never overwrite semantic-convention keys like service.name, service.namespace, or trace_id with a different type — write derived values to new attribute names.
• Validate config with otelcol-contrib validate and test attribute/transform changes against a staging backend before production.
• Keep a debug exporter available (behind a staging pipeline or feature flag) so malformed payloads are inspectable.
• Match exporter protocol to endpoint: gRPC → 4317, HTTP/proto → 4318/v1/*; don’t mix them.
• Pin and track your Collector and backend OTLP versions so a proto field the backend rejects is caught at upgrade time.
• Alert on otelcol_exporter_send_failed_spans with low queue size — the signature of permanent, non-retried drops.

Quick Command Reference

# Find permanent InvalidArgument drops in Collector logs
journalctl -u otelcol-contrib -f | grep -i 'InvalidArgument\|Permanent error'

# Validate the Collector config after editing processors
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

# Confirm the gRPC endpoint and protocol are reachable
grpcurl -plaintext ingest.example.com:4317 list

# Watch permanent-drop counters (send_failed climbs, queue stays low)
curl -s http://localhost:8888/metrics | grep -E 'exporter_send_failed|exporter_queue_size'

Conclusion

Permanent error: rpc error: code = InvalidArgument is the backend telling you the data, not the connection, is wrong — and because it is permanent, the Collector drops it without retry. Read the desc field for the exact offending attribute or field, reproduce the payload with a debug exporter, and fix the type, the protocol, or the processor that mangled it. Since retries won’t save malformed data, catching these in staging with config validation and a debug pipeline is the only reliable defense against silent, service-specific telemetry gaps.

Related

• OpenTelemetry Error Guide: ‘partial success’ spans dropped
• OpenTelemetry Error Guide: ‘no more retries left; dropping data’
• OpenTelemetry Error Guide: ‘ResourceExhausted: received message larger than max’