🐰 RabbitMQ Error: rabbitmq_prometheus /metrics Scrape Timeout Under Load

Exact Error Message

# From Prometheus targets page / logs:
level=warn msg="Error on ingesting samples" scrape_pool=rabbitmq
  target=http://rabbit-01:15692/metrics err="context deadline exceeded"

# rabbitmq_prometheus target shown as DOWN:
Get "http://rabbit-01:15692/metrics": context deadline exceeded
  (Client.Timeout exceeded while awaiting headers)

A manual scrape confirms the endpoint is simply too slow to answer within the scrape window:

$ time curl -s http://rabbit-01:15692/metrics > /dev/null
real    0m14.812s

What It Means

The rabbitmq_prometheus plugin exposes broker metrics on port 15692 at /metrics. Prometheus scrapes that endpoint on a fixed interval and gives up after scrape_timeout. The timeout means RabbitMQ did not finish rendering the metrics response before Prometheus’s deadline, so the target flips to DOWN and you lose visibility exactly when the broker is busiest.

The default /metrics endpoint returns aggregated metrics, which is fast. But if you scrape the per-object endpoint (/metrics/per-object) or run with per-object metrics enabled on a broker that has tens of thousands of queues, channels, and connections, RabbitMQ has to serialize a metric line for every object on every scrape. Under load that rendering work competes with message processing and blows past the scrape timeout.

Common Causes

• Scraping /metrics/per-object (or prometheus.return_per_object_metrics = true) on a broker with a very high object count.
• A large number of queues/connections/channels so aggregation itself becomes expensive.
• scrape_timeout set too low (default 10s) relative to how long a big broker takes to render.
• The node is CPU-saturated, so metric collection is starved by message traffic.
• Scraping too frequently (short scrape_interval) so a slow scrape overlaps the next one.
• A single Prometheus scraping many large brokers without spreading the load.

Diagnostic Commands

Time a manual scrape from the Prometheus host to see the real render latency:

time curl -s http://rabbit-01:15692/metrics > /dev/null

Compare aggregated vs per-object cost:

time curl -s http://rabbit-01:15692/metrics/per-object > /dev/null

Check how many objects the broker is exposing (this drives render time):

rabbitmqctl list_queues --quiet | wc -l
rabbitmqctl list_connections --quiet | wc -l
rabbitmqctl list_channels --quiet | wc -l

Confirm the plugin is enabled and listening:

rabbitmq-plugins list -e | grep prometheus
ss -ltnp | grep 15692

Step-by-Step Resolution

1. Determine which endpoint Prometheus targets. If it is /metrics/per-object, switch to the aggregated /metrics endpoint unless you genuinely need per-queue series:

# prometheus.yml
scrape_configs:
  - job_name: rabbitmq
    metrics_path: /metrics          # aggregated, fast
    static_configs:
      - targets: ["rabbit-01:15692"]

2. Ensure per-object metrics aren’t forced on globally. Leave this off unless required:

# /etc/rabbitmq/rabbitmq.conf
prometheus.return_per_object_metrics = false

3. Raise the scrape timeout and interval so a large broker has room to answer, and keep timeout well under interval:

scrape_configs:
  - job_name: rabbitmq
    scrape_interval: 30s
    scrape_timeout: 25s
    metrics_path: /metrics
    static_configs:
      - targets: ["rabbit-01:15692"]

4. If you need per-queue detail, scrape it as a separate, less-frequent job so it never blocks the core aggregated scrape:

  - job_name: rabbitmq_detailed
    scrape_interval: 60s
    scrape_timeout: 50s
    metrics_path: /metrics/detailed
    params:
      family: ["queue_coarse_metrics"]
    static_configs:
      - targets: ["rabbit-01:15692"]

5. Reduce object churn where possible — remove abandoned exclusive/auto-delete queues and short-lived connections that inflate the series count:

rabbitmqctl list_queues name messages consumers | sort -k2 -n | tail

6. Reload Prometheus and confirm the target is UP and the scrape now completes in time:

curl -s -X POST http://localhost:9090/-/reload
time curl -s http://rabbit-01:15692/metrics > /dev/null

real    0m1.204s

Prevention

• Prefer the aggregated /metrics endpoint; only use per-object metrics for targeted, low-frequency scraping.
• Keep scrape_timeout comfortably below scrape_interval so a slow scrape never overlaps the next.
• Use the /metrics/detailed endpoint with a family filter when you need specific series instead of everything.
• Cap object growth: prune unused queues, and avoid connection/channel churn that inflates the series count.
• Scale out scraping — one Prometheus per region or sharded targets — rather than one instance scraping every large broker.
• Alert on the up metric for the RabbitMQ job so a flapping target is caught immediately.

Related Errors

• context deadline exceeded from any exporter — the generic Prometheus scrape-timeout signal, not RabbitMQ-specific.
• connection.blocked / resource alarms — a broker under memory/disk pressure will also render metrics slowly.
• too many channels / channel leaks — channel explosions inflate per-object metric counts and render time.
• management UI slow to load — the same object-count pressure affecting the management plugin.

Frequently Asked Questions

Should I use /metrics or /metrics/per-object? Use aggregated /metrics for routine scraping; /metrics/per-object produces one series per queue/connection/channel and is far slower on large brokers.

What scrape_timeout should I set? Set it below your scrape_interval with headroom — for a large broker, a 30s interval with a 25s timeout is a safe starting point, then tune from the measured render time.

How do I get per-queue metrics without timing out? Scrape the /metrics/detailed endpoint with a family filter in a separate, less-frequent job so detailed collection never blocks the core scrape.

Why did it only start failing at scale? Per-object rendering cost grows with the number of queues, connections, and channels, so a broker that was fine at hundreds of objects times out at tens of thousands. Describe your object counts in the DevOps AI prompt library for a tuned scrape config, or browse more RabbitMQ guides.