🚨 Prometheus Error Guide: 'metrics endpoint 404' — Fix a Scrape Target Not Found

Overview

Prometheus reports a target as down and the Status > Targets page shows the scrape failing with an HTTP 404. The error text on the target reads:

server returned HTTP status 404 Not Found

Curling the path Prometheus is scraping returns the same thing:

$ curl -s -o /dev/null -w '%{http_code}\n' http://node-1:9100/metrics
404

The target is reachable — the TCP connection and HTTP response succeed — but the path Prometheus asked for does not exist on that server.

Symptoms

• Status > Targets shows the target as DOWN with server returned HTTP status 404 Not Found.
• The up metric for that target is 0.
• curl http://<target>:<port>/metrics returns 404, not a metrics payload.
• Other targets in the same job scrape fine, or the target worked before a config change.
• A reverse proxy or ingress sits in front of the exporter.

Symptoms vs. Root Causes

• Wrong metrics_path — the exporter serves metrics at a non-default path (e.g. /actuator/prometheus, /api/v1/metrics) but the job uses the default /metrics.
• Wrong port — scraping the app’s main port instead of the exporter’s metrics port.
• Wrong scheme — hitting http when the endpoint requires https, or a proxy that only exposes one.
• Reverse-proxy path prefix — an ingress strips or adds a prefix (/api/metrics) that the scrape config does not account for.
• Exporter not enabled — the app has a Prometheus endpoint that is disabled or gated behind a feature flag.
• Typo or trailing slash — metrics_path: /metrics/ when the server only answers /metrics.

Diagnostic Workflow

Confirm the failure straight from the shell, exactly as Prometheus sees it. Print the status code and, if 404, the body:

curl -s -o /dev/null -w '%{http_code}\n' http://node-1:9100/metrics
curl -s http://node-1:9100/metrics | head

If that returns 404, probe likely alternative paths for the exporter or framework:

for p in /metrics /actuator/prometheus /api/v1/metrics /prometheus; do
  code=$(curl -s -o /dev/null -w '%{http_code}' "http://node-1:9100${p}")
  echo "$code  $p"
done

Look at the scrape config and fix whichever field is wrong. All three of metrics_path, port (in targets), and scheme matter:

# prometheus.yml
scrape_configs:
  - job_name: api
    metrics_path: /actuator/prometheus   # not the default /metrics
    scheme: http
    static_configs:
      - targets: ['node-1:9100']         # exporter port, not the app port

Check the target’s live state and error string in the UI’s targets API rather than guessing:

curl -s http://node-1:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job=="api") | {url: .scrapeUrl, health: .health, err: .lastError}'

Confirm in PromQL whether the target is up, and read the failure reason:

up{job="api"} == 0

If a reverse proxy is involved, curl through it the same way Prometheus does, following any redirect, so you can see whether the proxy or the exporter returns the 404:

curl -v http://node-1:9100/metrics 2>&1 | grep -E '^< HTTP|Location'

After correcting the config, validate and reload:

promtool check config prometheus.yml
curl -s -X POST http://node-1:9090/-/reload

Example Root Cause Analysis

A Spring Boot service was added to the api job and immediately showed DOWN with server returned HTTP status 404 Not Found. up{job="api"} was 0 for that instance while the older instances were 1. Curling http://node-1:9100/metrics returned 404, confirming the path, not connectivity, was the issue.

Spring Boot’s Actuator exposes Prometheus metrics at /actuator/prometheus, not /metrics. Looping over candidate paths with curl found the metrics payload at /actuator/prometheus returning 200. The scrape job had inherited the default metrics_path: /metrics. Setting metrics_path: /actuator/prometheus on the job, validating with promtool check config, and hot-reloading brought the target to UP on the next scrape, and up{job="api"} flipped to 1.

Prevention Best Practices

• Verify a new target with curl http://<target>:<port>/metrics before adding it to a scrape job.
• Document each exporter’s real metrics_path — Actuator, kube-state-metrics, and app frameworks all differ from the default.
• Scrape the exporter/metrics port explicitly; do not assume it matches the app’s service port.
• Keep scheme correct and consistent; if a proxy terminates TLS, scrape the backend directly where possible.
• Account for reverse-proxy path prefixes in metrics_path, and test through the proxy with curl -v.
• Alert on up == 0 so a 404 target is caught immediately. See /dashboard/monitoring-alerts/.

Quick Command Reference

# Reproduce the 404 exactly as Prometheus sees it
curl -s -o /dev/null -w '%{http_code}\n' http://node-1:9100/metrics

# Read the live scrape URL and last error
curl -s http://node-1:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job=="api") | {url:.scrapeUrl, err:.lastError}'

# Validate and reload after fixing metrics_path/port/scheme
promtool check config prometheus.yml
curl -s -X POST http://node-1:9090/-/reload

up{job="api"} == 0

Conclusion

A 404 means the target is reachable but the path is wrong — this is a configuration mismatch, not a network outage. Reproduce it with curl, discover the exporter’s real metrics_path, and confirm the port and scheme match reality. Fix the scrape config, validate with promtool, reload, and watch up return to 1 on Status > Targets. More scrape-config patterns are in the Prometheus stack guide.