🚨 OpenTelemetry Error Guide: 'partial success' spans dropped — Stop Silent Data Loss

Overview

OTLP defines a “partial success” response: the export call succeeds (HTTP 200 / gRPC OK), but the backend reports that it accepted only some of the items and rejected the rest. The SDK or Collector logs it as a warning, which is easy to miss:

2026-07-09T08:52:41.330Z	warn	otlpreceiver/otlp.go:132	Partial success response from OTLP export	{"kind": "exporter", "name": "otlp", "rejected_spans": 128, "error_message": "spans dropped: attribute value exceeds max length"}

The OTLP response body carries the detail in partial_success with rejected_spans (or rejected_data_points/rejected_log_records) and a human-readable error_message. A 200/OK at the transport layer does not mean every span was stored.

Symptoms

• Trace counts in your backend are lower than what services emit, with no export errors.
• Logs contain Partial success response warnings with a non-zero rejected_spans.
• The error_message mentions bad attributes, timestamps, resource limits, or unsupported fields.
• Only certain spans/services are missing — the ones tripping the backend’s validation.
• Overall exports report success, so naive success-rate dashboards look healthy.

Common Root Causes

• Attribute value too long — an attribute exceeds the backend’s max length or count limit.
• Invalid timestamps — start/end times out of range, negative durations, or clock skew.
• Unsupported attribute types — nested/complex values the backend cannot store.
• Rate/quota limits — the backend accepts up to a quota and rejects the overflow as partial.
• Malformed resource — missing required resource fields or oversized resource attributes.
• Cardinality guards — the backend drops items that would exceed a cardinality cap.

Diagnostic Workflow

Turn up exporter/receiver logging so partial-success detail is visible, and add the debug exporter to inspect rejected items locally:

exporters:
  otlp:
    endpoint: backend.example.com:4317
  debug:
    verbosity: detailed

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp, debug]

Bound attribute sizes with SDK environment variables so oversized values are truncated before export instead of rejected:

export OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT=4096
export OTEL_ATTRIBUTE_COUNT_LIMIT=128
export OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT=128

Grep the Collector logs for partial-success warnings and read the backend’s reason:

journalctl -u otelcol-contrib --since '30 min ago' | grep -i 'partial success\|rejected_spans\|error_message'

Optionally enforce limits in the pipeline with the transform/attributes processor so bad data is normalized centrally:

processors:
  attributes:
    actions:
      - key: db.statement
        action: update
        value: "[redacted]"

Example Root Cause Analysis

A service annotated spans with a full SQL statement as db.statement. Some statements were tens of kilobytes, and the backend enforced a 4KB attribute-value limit. On export the transport returned OK, but the OTLP partial_success reported rejected_spans with error_message: "attribute value exceeds max length". Because transport succeeded, dashboards showed a healthy export rate while ~5% of spans quietly vanished.

The fix worked at two layers. In the SDK, OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT=4096 truncated oversized values before export so they satisfied the backend limit. As defense in depth, the Collector’s attributes processor redacted db.statement to a bounded placeholder, keeping the span (and its timing/trace linkage) while dropping the oversized payload. rejected_spans fell to zero and trace counts matched emission.

Prevention Best Practices

• Treat partial-success warnings as errors: alert on non-zero rejected_spans/rejected_data_points.
• Set SDK attribute limits (OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT, *_COUNT_LIMIT) to match backend limits.
• Avoid putting unbounded data (raw SQL, request bodies, stack traces) in attributes; truncate or redact.
• Validate clocks (NTP) so timestamps stay within the backend’s accepted range.
• Normalize attributes centrally with the Collector’s transform/attributes processor.
• Reconcile emitted vs stored span counts periodically to catch silent partial loss.

Quick Command Reference

# Surface partial-success detail
journalctl -u otelcol-contrib | grep -i 'partial success\|rejected'

# SDK attribute limits currently in effect
env | grep -E 'OTEL_.*ATTRIBUTE.*LIMIT'

# Inspect rejected items locally via the debug exporter
journalctl -u otelcol-contrib | grep -iA3 'ResourceSpans'

# Confirm host clock is synced
timedatectl status | grep -i 'synchronized'

Conclusion

OTLP “partial success” is the trap where a 200/OK hides real data loss — the backend accepted some items and rejected others, reporting the count and reason in partial_success. The fix is to make the rejection visible (alert on rejected_spans) and to keep exports within backend limits: bound attribute sizes in the SDK, redact unbounded values in the Collector, and keep clocks synced. Once partial-success warnings are treated as first-class errors, silent gaps in your traces become a solvable, observable problem.