🚨 Loki Error Guide: 'max concurrent tail requests limit exceeded' — Manage Live Tail Connections

Overview

Loki refuses to open a new live-tail stream when the number of simultaneous tail connections for a tenant is already at its cap. The tail endpoint (used by Grafana’s “Live” mode and logcli query --tail) returns an error like this:

max concurrent tail requests limit exceeded, count > limit (10 > 10)

Live tailing holds an open, long-lived connection from the querier back to the ingesters so new matching lines can be streamed in near-real-time. Because each tail consumes a persistent connection and ingester resources, Loki caps how many can run at once per tenant with limits_config.max_concurrent_tail_requests (default 10). When the cap is reached, additional GET /loki/api/v1/tail requests are rejected until an existing tail closes. This is a concurrency guardrail, not a data problem — no logs are lost, you simply cannot open another live stream right now.

Symptoms

• Grafana Explore in “Live” mode fails to start tailing with an error, while ordinary range queries work fine.
• logcli query --tail exits immediately with max concurrent tail requests limit exceeded.
• The message shows (count > limit) with equal or near-equal numbers, e.g. (10 > 10).
• The problem clears on its own after users close browser tabs or a few minutes pass.
• A team with many engineers all live-tailing the same busy service hits it repeatedly.

Common Root Causes

• Too many people live-tailing at once — each open Grafana “Live” panel is one tail.
• Leaked/zombie tail connections — browser tabs left open or clients that did not close cleanly keep tails alive until they time out.
• A low default max_concurrent_tail_requests relative to team size and habits.
• Dashboards or kiosks left in Live mode permanently holding tails.
• Scripts that open a tail per run without closing the previous one.

How to diagnose

1. Read the counts in the error — (count > limit). If count equals limit, every slot is genuinely in use; the question is whether those tails are active or leaked.

2. Confirm the configured cap:

   curl -s http://loki.internal:3100/config \
     | grep -i max_concurrent_tail_requests

3. Find who is tailing — check querier logs for tail request paths and the tenant header:

   kubectl logs -l app=loki,component=querier --tail=200 \
     | grep -i '/loki/api/v1/tail'

4. Distinguish active from leaked tails — ask the team who is live-tailing right now. If far fewer people are actively tailing than the count suggests, the slots are held by stale connections that have not timed out yet.

5. Check for permanent Live dashboards — a wall display or a pinned Explore tab in Live mode holds a slot indefinitely.

Fixes

Close stale tails first — have users exit Grafana “Live” mode and close abandoned Explore tabs; kill any lingering logcli --tail processes. Slots free up as connections drop.

Prefer bounded range queries over live tail for most debugging — a short auto-refreshing range query gives near-live results without holding a persistent tail connection:

# Instead of --tail, poll a small recent window
logcli query '{app="checkout"}' --since 2m --limit 200

Raise the cap deliberately for a tenant whose team legitimately needs more concurrent tails, as a scoped override rather than a global change:

limits_config:
  max_concurrent_tail_requests: 10   # global default

# per-tenant override (runtime overrides file):
# overrides:
#   platform-team:
#     max_concurrent_tail_requests: 25

Tighten tail timeouts so leaked connections reclaim their slot sooner — ensure any reverse proxy in front of the tail endpoint does not hold idle websocket/streaming connections open longer than necessary.

Educate on Live mode — discourage leaving dashboards and kiosks pinned in Live mode; use a normal refresh interval instead so tail slots stay available for active debugging.

What to watch out for

• Each live tail consumes an ingester connection; a high max_concurrent_tail_requests across many tenants adds sustained load on ingesters, so raise it per tenant rather than globally.
• Leaked tails often outnumber active ones — the fix is usually closing stale connections, not raising the limit. Bumping the cap to hide leaks just delays the problem.
• Live tailing bypasses the results and chunk caches, so heavy tailing on a busy stream is comparatively expensive; steer routine debugging to range queries.
• A proxy or load balancer that does not support streaming/websockets cleanly can leave half-open tails that keep counting against the limit — verify the proxy closes dead connections promptly.
• Tail results are best-effort real-time and can miss lines during ingester churn; do not rely on live tail for completeness, use a range query to confirm.

Related

• Loki Error Guide: ‘context deadline exceeded’ — the other long-lived read-path connection failure and its timeout stack.
• Loki Error Guide: ‘too many outstanding requests’ — the read-path concurrency limit for ordinary queries.
• Loki Error Guide: ‘empty ring’ — when tails and queries fail because ingesters are missing from the ring.