🚨 Grafana Error Guide: 'status code 429 Too Many Requests' — Fix Datasource Rate Limiting

Overview

Grafana surfaces this when a datasource backend (Prometheus, Loki, Mimir, or a cloud metrics API) rejects a query because the client has exceeded a rate or concurrency limit. Panels go red and the response is:

the request was rejected with status code 429 (Too Many Requests)

Backend-specific variants carry the same 429 status:

429 Too Many Requests: too many outstanding requests

ThrottlingException: Rate exceeded (status code: 429)

This is not a Grafana bug — the backend is deliberately shedding load. The fix is to reduce how much Grafana asks for, spread it out, or raise the backend limit where that’s safe.

Symptoms

• Panels intermittently show 429 Too Many Requests; a manual refresh sometimes works, sometimes fails.
• Failures spike on the hour or when many users open the same heavy dashboard at once.
• Only one datasource is affected; others on the same Grafana are fine.
• TV/kiosk dashboards with short auto-refresh fail most often.
• The backend’s own metrics show request/concurrency limits being hit at the same timestamps.

Common Root Causes

• Aggressive dashboard auto-refresh (5s/10s) multiplied across many panels and many concurrent viewers.
• Panel/variable fan-out — a repeat panel or an includeAll template variable issuing dozens of queries per refresh.
• Backend concurrency caps — Prometheus --query.max-concurrency, Thanos/Mimir -querier.max-concurrent, or a query frontend queue limit.
• Cloud API quotas — CloudWatch GetMetricData, Azure Monitor, or Google Cloud Monitoring per-second request limits.
• No query caching, so identical panels from many users each hit the backend instead of a cached result.
• A stampede at the top of the minute when many dashboards refresh on the same aligned schedule.

Diagnostic Workflow

1. Confirm which datasource and when. Read Grafana logs for the 429s and correlate timestamps:

journalctl -u grafana-server --since '30 min ago' | grep -i '429\|too many requests'

2. Find the heavy dashboards. Identify boards with short refresh and high panel/variable fan-out via the API:

# List dashboards and their refresh setting
for uid in $(curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/search?type=dash-db | jq -r '.[].uid'); do
  curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
    http://localhost:3000/api/dashboards/uid/$uid \
    | jq -r '.dashboard | "\(.title): refresh=\(.refresh) panels=\(.panels|length)"'
done

3. Check the backend’s rate/concurrency limits. For Prometheus/Mimir:

# Prometheus concurrency and queue flags
curl -s http://prometheus:9090/api/v1/status/flags | jq '."query.max-concurrency"'

# Mimir/Thanos query-frontend queue metric
curl -s http://prometheus:9090/metrics | grep -i 'engine_queries_concurrent\|query_frontend_queue_length'

4. Reduce Grafana-side load first. Lower auto-refresh, cap variables, and set a minimum refresh in grafana.ini:

[dashboards]
min_refresh_interval = 30s

5. Enable query caching (Grafana Enterprise/Cloud) so identical queries from many viewers hit a cache, not the backend:

[caching]
enabled = true

[caching.memory]
default_ttl = 60s

Or, per-datasource cache TTL via provisioning jsonData.cacheDurationSeconds.

6. Raise the backend limit only if it’s genuinely under-provisioned. For Prometheus, increase --query.max-concurrency; for cloud datasources, request a quota increase — but treat this as the last lever, after refresh and caching.

Example Root Cause Analysis

An office wall of six kiosk dashboards, each auto-refreshing every 5 seconds against a single Prometheus, started throwing 429 Too Many Requests every morning as engineers also opened their own dashboards. Prometheus was healthy on CPU but --query.max-concurrency was the default 20.

The API scan showed each kiosk board had 15+ panels plus an includeAll instance variable, so a single refresh issued ~40 concurrent queries; six boards at 5s alignment produced bursts far above 20 concurrent. The 429s were the query engine shedding the overflow.

Fix: raised the kiosk refresh to 30s, set min_refresh_interval = 30s globally, replaced includeAll with a top-N instance variable, and enabled a 60s query cache. Concurrent queries dropped below the limit and the 429s stopped — Prometheus’s concurrency setting was never the real bottleneck, the fan-out was.

Prevention Best Practices

• Set min_refresh_interval so no one can pin a dashboard to 5s auto-refresh against a shared backend.
• Avoid includeAll/repeat fan-out on high-cardinality variables; use top-N or explicit selections.
• Enable query caching so identical panels across viewers share one backend hit.
• Stagger refresh schedules rather than letting every board fire at the top of the minute.
• Monitor backend concurrency/queue depth and alert before it saturates, so 429s never reach users.
• Right-size backend limits and cloud quotas deliberately, only after Grafana-side load is under control.

Quick Command Reference

# Find 429s in Grafana logs
journalctl -u grafana-server --since '30 min ago' | grep -i 'too many requests'

# Audit dashboard refresh + panel counts
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" http://localhost:3000/api/search?type=dash-db | jq '.[].uid'

# Check Prometheus concurrency flag
curl -s http://prometheus:9090/api/v1/status/flags | jq '."query.max-concurrency"'

# Test a datasource's health
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" http://localhost:3000/api/datasources/uid/$UID/health | jq

Conclusion

A 429 from a Grafana datasource means the backend is intentionally limiting load, not that Grafana is broken. Identify the datasource and timing, find the dashboards with short refresh and query fan-out, and fix the demand side first — raise min_refresh_interval, cap variables, and enable caching. Only raise backend concurrency or cloud quotas once the Grafana-side stampede is tamed, or you’ll just shed load at a higher ceiling.