🚨 Loki Error Guide: 'Per stream rate limit exceeded' — Raise Stream Limits and Fix Label Cardinality

Overview

Loki’s distributor rejects log lines with this error when a single stream pushes data faster than its configured per-stream rate limit. Promtail, Grafana Alloy, or the Loki push API surfaces it as an HTTP 429:

Ingester.Push: rpc error: code = Code(429) desc = Per stream rate limit exceeded (limit: 3MB/sec) while attempting to ingest for stream '{app="checkout"}' totaling 5MB, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased

A stream in Loki is one unique combination of label values. Loki enforces a per-stream throughput cap (per_stream_rate_limit) with a short burst allowance (per_stream_rate_limit_burst). When one stream exceeds that rate, the distributor drops the excess lines and returns 429. The limit exists deliberately: it stops a single noisy source from overwhelming an ingester. Hitting it means either the limit is too low for legitimate volume, or too many logs are collapsed into one stream.

Symptoms

• Promtail/Alloy logs fill with 429 Per stream rate limit exceeded and final error sending batch.
• Gaps appear in Grafana for one specific app while other apps log normally.
• The error names a specific stream selector, e.g. stream '{app="checkout"}'.
• loki_discarded_samples_total with reason per_stream_rate_limit increases.
• Clients back off and retry, so logs arrive late or are lost entirely under sustained load.

Common Root Causes

• A genuinely chatty stream — one app or pod logging above the per-stream cap.
• Under-labeled streams — everything from a service collapsed into a single stream because no distinguishing label (pod, instance) is attached.
• Default limit too low — per_stream_rate_limit defaults around 3–4MB/s, below a busy service’s peak.
• Burst too small — spiky logging exceeds per_stream_rate_limit_burst even when the average is fine.
• A logging loop or stack-trace storm temporarily flooding one stream.

Diagnostic Workflow

Confirm which streams are being throttled by watching discard metrics on the distributor:

curl -s http://loki-distributor:3100/metrics \
  | grep 'loki_discarded_samples_total.*per_stream_rate_limit'

Find the volume of a suspect stream with a metric query in Grafana or logcli:

# Bytes per second for the hot stream, last hour
logcli instant-query 'sum(bytes_rate({app="checkout"}[1m]))'

Read the effective limits from config (global and per-tenant overrides):

limits_config:
  per_stream_rate_limit: 5MB        # sustained bytes/sec per stream
  per_stream_rate_limit_burst: 20MB # short-term burst allowance
  # per-tenant overrides live in the runtime overrides file:
# overrides:
#   checkout-team:
#     per_stream_rate_limit: 10MB
#     per_stream_rate_limit_burst: 40MB

If one stream carries traffic from many pods, the fix is a label change, not a limit change — check whether a pod or instance label is missing from the client scrape config.

Example Root Cause Analysis

A checkout service logged verbose request traces. Its Promtail config attached only {app="checkout"}, so all 12 pods wrote into one Loki stream. Individually each pod produced 0.5MB/s, but combined they pushed 6MB/s into a single stream against a 3MB/s limit. The distributor dropped roughly half the lines and Grafana showed jagged, incomplete logs.

Rather than just raising the limit, the team added the pod label in the Promtail relabel_configs. That split the traffic into 12 separate streams of ~0.5MB/s each — every one comfortably under the 3MB/s cap — so no discards remained. They also nudged per_stream_rate_limit to 5MB and per_stream_rate_limit_burst to 20MB as headroom for future spikes. Log loss went to zero without adding cardinality risk, because pod was a bounded, meaningful label.

Prevention Best Practices

• Label streams so throughput naturally spreads across pods/instances, keeping any single stream under the cap.
• Avoid unbounded labels (request IDs, user IDs) that explode cardinality while chasing rate limits.
• Set per_stream_rate_limit and per_stream_rate_limit_burst to match real peak volume, with burst covering spikes.
• Use per-tenant overrides so a noisy team gets more room without loosening limits globally.
• Reduce log verbosity at the source; sample or drop debug lines in the pipeline before they reach Loki.
• Alert on loki_discarded_samples_total{reason="per_stream_rate_limit"} so drops are caught before users notice gaps.

Quick Command Reference

# Discards attributed to per-stream limiting
curl -s localhost:3100/metrics | grep per_stream_rate_limit

# Measure a stream's byte rate
logcli instant-query 'sum(bytes_rate({app="checkout"}[1m]))'

# Inspect running limits
curl -s http://loki:3100/config | grep -A2 per_stream_rate_limit

limits_config:
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 20MB

Conclusion

Per stream rate limit exceeded protects Loki from a single overloaded stream, and the message tells you exactly which selector is affected. The best fix is usually to split that stream with a bounded label like pod so traffic spreads out, then set per_stream_rate_limit and its burst to match real peak volume. Reserve broad limit increases for genuinely high-throughput services, use per-tenant overrides, and alert on discards so dropped logs never go unnoticed.