🚨 Prometheus Error Guide: 'metrics endpoint 500' — Fix an Exporter Failing to Render

Overview

The target is reachable and the path is correct, but the application or exporter throws while generating its metrics, so /metrics returns HTTP 500 and Prometheus has nothing to scrape. Status > Targets shows:

server returned HTTP status 500 Internal Server Error

Curling the endpoint returns an error body from the client library instead of metrics:

$ curl -s http://node-1:9100/metrics
An error has occurred while serving metrics:

collected metric "db_pool_connections" was collected before with the same name and label values

Unlike a 404 (wrong path), a 500 means the endpoint exists but the code that renders it failed.

Symptoms

• Status > Targets shows DOWN with server returned HTTP status 500 Internal Server Error.
• The up metric for the target is 0.
• curl http://<target>:<port>/metrics returns a stack trace or a client-library error string, not metrics.
• The exporter or application logs show an exception each time Prometheus scrapes.
• The endpoint sometimes returns 200 and sometimes 500 (an intermittent/failing collector).

Common Root Causes

• A failing custom collector — a Collect()/collect() implementation raises when a downstream call (DB, API) errors or times out.
• Duplicate registration — a metric registered twice, or the same name/label set emitted more than once (a registry error).
• Client-library gather errors — inconsistent label dimensions on a metric family, causing the gatherer to abort the whole page.
• A slow collector hitting the scrape timeout — the handler exceeds scrape_timeout, so the response is cut off or errors.
• Unhandled exception in the app’s metrics handler — a nil map, panic, or serialization failure during render.
• Resource exhaustion — the process is out of memory/file descriptors and cannot build the response.

Diagnostic Workflow

Reproduce with a verbose curl so you see the status line and the error body the library returned:

curl -v http://node-1:9100/metrics 2>&1 | sed -n '1,40p'

The body usually names the failure directly. Cross-reference it with the exporter/application logs at the moment of the scrape:

journalctl -u node-exporter --since '10 min ago' | grep -iE 'error|panic|collect|metrics'
# or, in Kubernetes
kubectl logs deploy/api -c metrics --since=10m | grep -iE 'error|panic|collect'

Confirm the target’s live error string and health from the Prometheus API:

curl -s http://node-1:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job=="api") | {health:.health, err:.lastError}'

If the error is a registry/duplicate problem, it is a code bug in how metrics are registered. The typical client-library messages are:

duplicate metrics collector registration attempted
collected metric ... was collected before with the same name and label values
inconsistent label cardinality

These mean a collector emitted the same series twice or with mismatched label sets in one gather. Fix the collector so each series is unique per scrape and every sample in a metric family carries the same label names.

If the endpoint is slow rather than throwing, the scrape timeout is the constraint. Compare how long the handler takes against the job’s scrape_timeout:

curl -s -o /dev/null -w 'time_total=%{time_total} code=%{http_code}\n' http://node-1:9100/metrics

# prometheus.yml — give a heavy exporter room, but fix the slow collector too
scrape_configs:
  - job_name: api
    scrape_interval: 30s
    scrape_timeout: 20s        # must be <= scrape_interval
    static_configs:
      - targets: ['node-1:9100']

Watch scrape health and duration in PromQL to catch intermittent 500s and timeouts:

up{job="api"} == 0
scrape_duration_seconds{job="api"} > 10

After a code or config fix, validate and reload:

promtool check config prometheus.yml
curl -s -X POST http://node-1:9090/-/reload

Example Root Cause Analysis

An in-house exporter began returning 500 for /metrics right after a deploy; up{job="api"} dropped to 0. curl -v http://node-1:9100/metrics returned collected metric "db_pool_connections" was collected before with the same name and label values, and the application log showed the same message on every scrape.

The new release added a per-request gauge that reused the db_pool_connections name already exposed by a background collector. On each gather, the client library saw two samples with identical name and labels and aborted the entire response — so no metrics at all were served, not just the offending one. The fix was to remove the duplicate registration and expose the pool gauge from a single collector. After redeploying, curl returned a clean 200 payload, scrape_duration_seconds was well under scrape_timeout, and the target returned to UP.

Prevention Best Practices

• Make custom collectors defensive: on a downstream error, log and skip that collector rather than raising and failing the whole /metrics page.
• Register each metric exactly once; guard against duplicate names and mismatched label cardinality in code review.
• Keep every sample in a metric family on the same label set to avoid gather-time inconsistency errors.
• Set scrape_timeout sensibly (and <= scrape_interval), but treat a collector near the timeout as a bug to fix, not a limit to raise.
• Add a synthetic check that curls /metrics in CI and fails the build on a non-200 response.
• Alert on up == 0 and rising scrape_duration_seconds so a failing render is caught fast. See /dashboard/monitoring-alerts/.

Quick Command Reference

# Reproduce and read the error body
curl -v http://node-1:9100/metrics 2>&1 | sed -n '1,40p'

# Correlate with exporter logs
journalctl -u node-exporter --since '10 min ago' | grep -iE 'error|panic|collect'

# Live target health and last error
curl -s http://node-1:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job=="api") | {health:.health, err:.lastError}'

# Measure render time vs scrape_timeout
curl -s -o /dev/null -w 'time_total=%{time_total} code=%{http_code}\n' http://node-1:9100/metrics

up{job="api"} == 0
scrape_duration_seconds{job="api"} > 10

Conclusion

A 500 on /metrics is a rendering failure inside the app or exporter, not a Prometheus problem — the endpoint exists but the code that builds it threw. Read the error body from curl -v, match it to the exporter logs, and fix the specific cause: a failing collector, a duplicate registration, inconsistent labels, or a collector slower than scrape_timeout. Validate, reload, and confirm up returns to 1. More exporter troubleshooting is in the Prometheus stack guide.