🚨 Loki Error Guide: 'Ingestion rate limit exceeded' — Raise or Rebalance the Tenant Rate Limit

Overview

Loki’s distributor rejects a push when a tenant exceeds its configured ingestion rate. Clients (Promtail, Alloy, OTel Collector) see an HTTP 429 with a body like this:

429 Too Many Requests
Ingestion rate limit exceeded for user tenant-a (limit: 4194304 bytes/sec) while attempting to ingest '1000' lines totaling '1048576' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased

The limit is enforced per tenant against the ingestion_rate_mb (sustained) and ingestion_burst_size_mb (short spikes) limits. Once tripped, the affected pushes are dropped and the sender must retry, so logs are delayed or lost if the client buffer overflows.

Symptoms

• Clients log repeated 429 responses and server returned HTTP status 429 Too Many Requests from loki.write/Promtail.
• Gaps or lag in dashboards for a specific tenant while other tenants are fine.
• loki_discarded_samples_total{reason="rate_limited"} and loki_discarded_bytes_total{reason="rate_limited"} climb.
• Distributor logs show Ingestion rate limit exceeded.
• A deploy, log-level change, or traffic spike immediately precedes the errors.

Common Root Causes

• Genuine volume growth — more pods, a new noisy service, or debug logging left on pushes the tenant over its sustained ingestion_rate_mb.
• Bursty senders — batched flushes exceed ingestion_burst_size_mb even though the average is under the sustained rate.
• A single hot tenant — multi-tenant cluster where one tenant’s default limit is too low for its real volume.
• Too-small defaults — the global limits_config defaults never tuned for production throughput.
• Distributor undercount of replicas — rate is divided across distributors; too few distributors or uneven load makes each reject earlier than expected.
• Retry storms — clients retrying rejected batches add load, deepening the rejection.

Diagnostic Workflow

Confirm which tenant and reason are driving the drops:

sum by (tenant, reason) (rate(loki_discarded_bytes_total[5m]))

Compare the tenant’s real ingest rate against its limit (bytes/sec):

sum by (tenant) (rate(loki_distributor_bytes_received_total[5m]))

Read the distributor logs for the exact limit and offending tenant:

kubectl logs -l app.kubernetes.io/component=distributor --since=15m \
  | grep -i "rate limit exceeded"

Inspect the effective per-tenant limits the runtime is using:

logcli --addr=http://loki-gateway/ config | grep -iE 'ingestion_rate|burst'

Check the current limits_config and any per-tenant overrides:

limits_config:
  ingestion_rate_strategy: local   # or global (divides across distributors)
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 6
  per_stream_rate_limit: 3MB
  per_stream_rate_limit_burst: 15MB

Per-tenant override in the runtime overrides file:

overrides:
  tenant-a:
    ingestion_rate_mb: 16
    ingestion_burst_size_mb: 24

Example Root Cause Analysis

A platform team saw tenant-a dashboards lag by minutes every deploy. loki_discarded_bytes_total{reason="rate_limited"} spiked to ~1 MB/s during each rollout while the steady-state rate stayed near 3 MB/s — under the 4 MB/s sustained limit. The trigger was rollout log bursts: new pods dumped startup logs simultaneously, briefly exceeding the 6 MB ingestion_burst_size_mb.

Rather than raise the sustained rate (which was fine), they increased ingestion_burst_size_mb to 24 MB for tenant-a via a per-tenant override to absorb deploy spikes, and enabled client-side backpressure so loki.write buffered instead of hammering retries. After the change, reason="rate_limited" drops fell to zero across deploys and the sustained rate was unaffected — confirming a burst-size problem, not a throughput problem.

Prevention Best Practices

• Set per-tenant overrides sized to each tenant’s real sustained and burst volume instead of relying on global defaults.
• Alert on rate(loki_discarded_bytes_total{reason="rate_limited"}[5m]) > 0 per tenant so you catch rejections before users notice gaps.
• Separate sustained (ingestion_rate_mb) from burst (ingestion_burst_size_mb) tuning — deploy spikes are usually a burst problem.
• Configure client-side backpressure/buffering (Alloy loki.write WAL, Promtail limits) so retries don’t amplify load.
• Understand ingestion_rate_strategy: with global, add distributors proportionally; with local, the limit is per-distributor.
• Keep debug logging off in production and cap chatty services at the source.

Quick Command Reference

# Drops by tenant and reason
# (run in Grafana Explore against the Loki datasource metrics, or Prometheus)
sum by (tenant, reason) (rate(loki_discarded_bytes_total[5m]))

# Actual received bytes/sec per tenant
sum by (tenant) (rate(loki_distributor_bytes_received_total[5m]))

# Distributor rejection logs
kubectl logs -l app.kubernetes.io/component=distributor --since=15m | grep -i "rate limit"

# Effective runtime limits
logcli --addr=http://loki-gateway/ config | grep -iE 'ingestion_rate|burst'

Conclusion

Ingestion rate limit exceeded is Loki protecting itself with a per-tenant throttle. Diagnose it by separating sustained rate from burst and by attributing the discarded bytes to a specific tenant and reason. Most incidents are either a genuine volume increase (raise ingestion_rate_mb), a deploy burst (raise ingestion_burst_size_mb), or a retry storm (add client backpressure). Fix the right one with a scoped per-tenant override, alert on discarded bytes, and cap noisy sources so the limit protects the cluster without dropping legitimate logs.