🚨 OpenTelemetry Error Guide: 'context canceled' on OTLP export — Fix Cancelled Exports

What this error means

This error appears when an in-flight OTLP export is aborted because the Go context driving it was cancelled before the request completed. It surfaces in application SDK logs and in Collector exporter logs:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Canceled desc = context canceled"}

The SDK-side variant, seen at process shutdown, reads:

traces export: Post "http://otel-collector:4318/v1/traces": context canceled

context canceled is distinct from context deadline exceeded: a deadline means time ran out, while cancelled means something actively tore the context down — usually a process shutting down mid-flush, a parent request being cancelled, or a client disconnecting. The data in that batch is dropped unless a retry queue re-sends it.

How the agent reports it

• Bursts of code = Canceled desc = context canceled at deploy, scale-down, or pod eviction time — but not during steady state.
• A short window of dropped spans/metrics coinciding with application or Collector restarts.
• Long-lived requests that get cancelled upstream also cancel the child export span’s context.
• The Collector logs context canceled on exports to a downstream gateway that is being rolled.
• Steady-state telemetry is fine; the errors cluster around lifecycle events.

Checking the agent configuration

Correlate the timestamps of the errors with lifecycle events. If they line up with deploys or evictions, it is a drain problem, not a network problem:

kubectl get events --field-selector reason=Killing --sort-by=.lastTimestamp | tail
kubectl logs deploy/my-app --previous | grep -i 'context canceled\|shutting down'

Check the graceful-shutdown budget on both sides. The pod needs enough grace period to flush, and the Collector’s sending_queue needs to drain:

kubectl get pod -l app=my-app -o jsonpath='{.items[0].spec.terminationGracePeriodSeconds}'

For custom code, confirm the exporter is NOT handed a request-scoped context. Background exporters must use context.Background(), not the incoming request’s ctx.

Enable a debug exporter on the Collector temporarily to see whether cancellations are inbound (clients disconnecting) or outbound (Collector cancelling downstream):

exporters:
  debug:
    verbosity: normal
service:
  telemetry:
    logs:
      level: debug

Agent configuration causes

• Shutdown race — the process (or Collector) begins terminating and cancels in-flight exports before the batch/queue has drained.
• Client disconnect — an HTTP/gRPC client cancels the parent request; if telemetry export inherits that request context, it is cancelled too.
• Rolling upgrade of the receiver — the downstream Collector/gateway closes the connection, and the client-side context is cancelled as the stream tears down.
• Aggressive graceful-shutdown timeout — terminationGracePeriodSeconds or the SDK’s shutdown timeout is too short to flush the last batch.
• Passing a request-scoped context to a background exporter in custom instrumentation, so ending the request kills the export.

Step-by-step resolution

Give the process time to flush on shutdown. Raise the pod grace period and add a preStop hook so traffic stops before the app exits:

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

Persist the Collector’s export queue so a restart resumes instead of cancelling in-flight batches:

extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue

exporters:
  otlp:
    endpoint: gateway:4317
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      storage: file_storage/queue

service:
  extensions: [file_storage/queue]

In the SDK, set an explicit shutdown timeout and force a final flush before exit. In Go, for example, call the provider’s Shutdown (or ForceFlush) with a fresh, generous context:

ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := tracerProvider.Shutdown(ctx); err != nil {
    log.Printf("otel shutdown: %v", err)
}

In custom instrumentation, never export on a request-scoped context. Detach it:

// Wrong: exportCtx := reqCtx  // cancelled when the request ends
exportCtx := context.WithoutCancel(reqCtx) // or context.Background()

Safer agent defaults

• A context canceled burst that is bounded to deploys is often acceptable data loss; alert only if it exceeds a small threshold or leaks into steady state.
• Persistent queues (file_storage) turn a shutdown cancellation into a resumed export — but only if the volume survives the restart; an emptyDir does not survive eviction.
• Do not confuse this with context deadline exceeded — that is a timeout/slowness problem and needs backend or timeout tuning, not drain tuning.
• Raising terminationGracePeriodSeconds too high slows rollouts; size it to your batch/flush time plus headroom, not arbitrarily large.
• If your app inherits export contexts from web-framework middleware, audit that the SDK export path is detached from request cancellation.

Related agent errors

• OpenTelemetry Error Guide: ‘context deadline exceeded’
• OpenTelemetry Error Guide: ‘sending queue is full’
• OpenTelemetry Error Guide: ‘No more retries left. Dropping data’