🚨 OpenTelemetry Error Guide: 'context canceled' on OTLP export — Fix Cancelled Exports

Overview

This error appears when an in-flight OTLP export is aborted because the Go context driving it was cancelled before the request completed. It surfaces in application SDK logs and in Collector exporter logs:

error	exporterhelper/queue_sender.go:128	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Canceled desc = context canceled"}

The SDK-side variant, seen at process shutdown, reads:

traces export: Post "http://otel-collector:4318/v1/traces": context canceled

context canceled is distinct from context deadline exceeded: a deadline means time ran out, while cancelled means something actively tore the context down — usually a process shutting down mid-flush, a parent request being cancelled, or a client disconnecting. The data in that batch is dropped unless a retry queue re-sends it.

Symptoms

• Bursts of code = Canceled desc = context canceled at deploy, scale-down, or pod eviction time — but not during steady state.
• A short window of dropped spans/metrics coinciding with application or Collector restarts.
• Long-lived requests that get cancelled upstream also cancel the child export span’s context.
• The Collector logs context canceled on exports to a downstream gateway that is being rolled.
• Steady-state telemetry is fine; the errors cluster around lifecycle events.

Common Root Causes

• Shutdown race — the process (or Collector) begins terminating and cancels in-flight exports before the batch/queue has drained.
• Client disconnect — an HTTP/gRPC client cancels the parent request; if telemetry export inherits that request context, it is cancelled too.
• Rolling upgrade of the receiver — the downstream Collector/gateway closes the connection, and the client-side context is cancelled as the stream tears down.
• Aggressive graceful-shutdown timeout — terminationGracePeriodSeconds or the SDK’s shutdown timeout is too short to flush the last batch.
• Passing a request-scoped context to a background exporter in custom instrumentation, so ending the request kills the export.

How to diagnose

Correlate the timestamps of the errors with lifecycle events. If they line up with deploys or evictions, it is a drain problem, not a network problem:

kubectl get events --field-selector reason=Killing --sort-by=.lastTimestamp | tail
kubectl logs deploy/my-app --previous | grep -i 'context canceled\|shutting down'

Check the graceful-shutdown budget on both sides. The pod needs enough grace period to flush, and the Collector’s sending_queue needs to drain:

kubectl get pod -l app=my-app -o jsonpath='{.items[0].spec.terminationGracePeriodSeconds}'

For custom code, confirm the exporter is NOT handed a request-scoped context. Background exporters must use context.Background(), not the incoming request’s ctx.

Enable a debug exporter on the Collector temporarily to see whether cancellations are inbound (clients disconnecting) or outbound (Collector cancelling downstream):

exporters:
  debug:
    verbosity: normal
service:
  telemetry:
    logs:
      level: debug

Fixes

Give the process time to flush on shutdown. Raise the pod grace period and add a preStop hook so traffic stops before the app exits:

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]

Persist the Collector’s export queue so a restart resumes instead of cancelling in-flight batches:

extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue

exporters:
  otlp:
    endpoint: gateway:4317
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      storage: file_storage/queue

service:
  extensions: [file_storage/queue]

In the SDK, set an explicit shutdown timeout and force a final flush before exit. In Go, for example, call the provider’s Shutdown (or ForceFlush) with a fresh, generous context:

ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := tracerProvider.Shutdown(ctx); err != nil {
    log.Printf("otel shutdown: %v", err)
}

In custom instrumentation, never export on a request-scoped context. Detach it:

// Wrong: exportCtx := reqCtx  // cancelled when the request ends
exportCtx := context.WithoutCancel(reqCtx) // or context.Background()

What to watch out for

• A context canceled burst that is bounded to deploys is often acceptable data loss; alert only if it exceeds a small threshold or leaks into steady state.
• Persistent queues (file_storage) turn a shutdown cancellation into a resumed export — but only if the volume survives the restart; an emptyDir does not survive eviction.
• Do not confuse this with context deadline exceeded — that is a timeout/slowness problem and needs backend or timeout tuning, not drain tuning.
• Raising terminationGracePeriodSeconds too high slows rollouts; size it to your batch/flush time plus headroom, not arbitrarily large.
• If your app inherits export contexts from web-framework middleware, audit that the SDK export path is detached from request cancellation.

Related

• OpenTelemetry Error Guide: ‘context deadline exceeded’
• OpenTelemetry Error Guide: ‘sending queue is full’
• OpenTelemetry Error Guide: ‘No more retries left. Dropping data’