🚨 Loki Error Guide: 'context canceled' — Trace Client Disconnects and Cancelled Sub-Queries

Overview

Loki logs this Go error whenever an operation is aborted because the context governing it was cancelled before the work finished:

context canceled

It commonly appears wrapped, for example:

level=error msg="error processing request" err="context canceled"

Unlike context deadline exceeded — which means a timeout fired — context canceled means something actively cancelled the request. Usually that “something” is the client going away: a user navigated off a dashboard, a browser tab closed, a proxy reset the connection, or the query frontend cancelled a sibling sub-query because the overall query already succeeded or failed. It is frequently benign log noise, but it can also mask a real problem (a proxy killing connections, or one shard failing and cancelling the rest). The skill is telling harmless cancellation from a genuine fault.

Symptoms

• Querier/frontend logs show context canceled bursts, often right after a user closes or refreshes a dashboard.
• The client saw a successful result, yet the server logged context canceled for related sub-queries.
• A reverse proxy (nginx/Envoy) logs client resets or upstream cancellations at the same timestamps.
• context canceled clusters around auto-refreshing dashboards or aborted logcli runs.
• Occasionally paired with a genuine failure where one sub-query errored and the frontend cancelled the rest.

Common Root Causes

• Client disconnects — user closes the tab, navigates away, or a dashboard auto-refresh supersedes the in-flight query.
• Proxy/load-balancer resets — a reverse proxy timeout or connection cap kills the client connection, cancelling the upstream context.
• Frontend cancelling sibling sub-queries — once a sharded query has what it needs (or one shard failed), the frontend cancels the remaining sub-queries; this is expected.
• logcli or a script aborted (Ctrl-C, killed process) mid-query.
• A genuine downstream failure cancelling the parent context, where context canceled is a symptom rather than the cause.

How to diagnose

1. Correlate with client behaviour — check whether the context canceled timestamps line up with dashboard refreshes, tab closes, or a user cancelling a query. If they do, it is almost certainly benign.

2. Separate cancellation from failure — search for a real error just before the cancellation. If a sub-query failed and then siblings were cancelled, the failure is the story, not the cancellation:

   kubectl logs -l app=loki,component=querier --tail=300 \
     | grep -B3 'context canceled'

3. Check the proxy layer — confirm a reverse proxy is not resetting connections early:

   grep -R 'proxy_read_timeout\|client_body_timeout\|keepalive' /etc/nginx/
   # Envoy: look for stream_idle_timeout / route timeout resets

4. Compare against context deadline exceeded — if the logs show deadlines rather than cancellations, that is a timeout problem with a different fix; do not conflate the two.

5. Measure impact — do users actually see failures, or only the server logs the message? If clients get correct results, the cancellations are cosmetic.

Fixes

Confirm it is benign and quiet the noise — if cancellations track client disconnects and users get correct results, treat it as expected. Avoid alerting on context canceled alone; it is not an error users experience.

Fix an over-eager proxy so it does not reset healthy long queries — align its timeouts and keepalive so the proxy is not the one cancelling:

# nginx in front of Loki: allow long queries to complete
proxy_read_timeout   310s;
proxy_send_timeout   310s;
proxy_http_version   1.1;
proxy_set_header     Connection "";

Tame dashboard refresh so a new auto-refresh does not fire before the previous query returns, which cancels the in-flight one — set the refresh interval comfortably above typical query latency, and narrow heavy panels so they finish quickly.

Chase the real error when cancellation follows a failure — if a sub-query or shard failed first, fix that (slow storage, a bad shard, an ingester issue). The context canceled on the siblings will stop once the root failure is resolved.

Filter the log noise if it drowns real errors — route benign context canceled lines to a lower severity in your log pipeline so genuine failures stay visible, rather than suppressing them blindly.

What to watch out for

• context canceled and context deadline exceeded are different: cancellation means someone aborted the request; deadline means it timed out. Raising timeouts will not fix a client disconnect.
• Do not page on context canceled by itself — it fires constantly from normal dashboard use and will cause alert fatigue. Alert on user-visible errors and latency instead.
• If cancellations spike without matching client activity, suspect the proxy or a failing shard cancelling the parent context, and dig into what happened just before.
• Aggressive dashboard auto-refresh is a common hidden source — each refresh can cancel the prior query, inflating the count harmlessly but noisily.
• Suppressing these lines entirely can hide the case where cancellation is the downstream symptom of a real failure; lower their severity, do not drop them.

Related

• Loki Error Guide: ‘context deadline exceeded’ — the timeout counterpart, with the full proxy-to-querier timeout stack.
• Loki Error Guide: ‘too many outstanding requests’ — read-path overload that can precede cancellations.
• Loki Error Guide: ‘the query time range exceeds the limit’ — bounding the heavy queries that get cancelled mid-flight.