🚨 OpenTelemetry Error Guide: 'Scrape failed' in the Prometheus receiver — Fix Metric Collection

Overview

This error appears when the Collector’s prometheus receiver cannot successfully scrape a target defined in its scrape_configs. The receiver logs one line per failed scrape, naming the scrape pool (the job_name), the target URL, and the underlying error:

Scrape failed  {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "otel-collector", "target": "http://10.0.0.20:8080/metrics", "error": "server returned HTTP status 500 Internal Server Error"}

The error field varies with the cause — a connection failure looks different from an HTTP status or a TLS problem:

Scrape failed  {"kind": "receiver", "name": "prometheus", "scrape_pool": "otel-collector", "target": "http://10.0.0.20:8080/metrics", "error": "Get \"http://10.0.0.20:8080/metrics\": dial tcp 10.0.0.20:8080: connect: connection refused"}

Scrape failed means the receiver reached the scrape step but could not retrieve a valid metrics response from the target — the metrics from that target are simply missing until the scrape succeeds.

Symptoms

• A specific target’s metrics are absent in your backend while other targets in the same receiver report normally.
• Collector logs repeat Scrape failed for one scrape_pool / target combination.
• The error field shows an HTTP status (500, 404, 401), a connection refused, or a TLS/x509 message.
• up{job="otel-collector"} is 0 for the affected target in the scraped output.
• The failure started after an app port change, a path rename, or a TLS/auth rollout on the target.

Common Root Causes

• Target down or wrong port — nothing is listening on the scraped host:port, yielding connection refused.
• Wrong metrics_path — the app exposes metrics at a non-default path (e.g. /actuator/prometheus) but the config scrapes /metrics.
• Application error — the target’s exporter returns 500/503 while rendering metrics, so the scrape gets a bad status.
• Missing auth — the endpoint requires a bearer token or basic auth the scrape_config does not supply, returning 401/403.
• Scheme/TLS mismatch — the target is HTTPS but scraped as http (or vice versa), or its certificate is not trusted.
• DNS or label rewrite — a relabel_config or bad hostname sends the scrape to the wrong or unresolvable address.

Diagnostic Workflow

Reproduce the exact scrape by hand from the Collector host to see the real status and body — this immediately separates “target is broken” from “config is wrong”:

# Same URL the receiver logged; -v shows status, redirects, and TLS
curl -v http://10.0.0.20:8080/metrics

# Confirm something is actually listening on that port
nc -zv 10.0.0.20 8080

# Follow the receiver's scrape failures live
journalctl -u otelcol-contrib -f | grep -i 'Scrape failed'

Fix the scrape_config so the scheme, port, metrics_path, and auth match what the target actually exposes. A correct receiver config looks like this:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 30s
          scrape_timeout: 10s
          metrics_path: /metrics        # match the app's real path
          scheme: http                  # or https, matching the target
          static_configs:
            - targets: ['10.0.0.20:8080']

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlphttp]

If the target requires authentication or TLS, add the corresponding fields — never inline a raw token, load it from a file or environment reference:

scrape_configs:
  - job_name: 'otel-collector'
    metrics_path: /metrics
    scheme: https
    authorization:
      type: Bearer
      credentials_file: /etc/otelcol/scrape-token   # token on disk, not in config
    tls_config:
      ca_file: /etc/otelcol/ca.crt
      insecure_skip_verify: false
    static_configs:
      - targets: ['10.0.0.20:8443']

Example Root Cause Analysis

A platform team migrated a Spring Boot service to expose metrics through Actuator at /actuator/prometheus, but the Collector’s prometheus receiver still scraped metrics_path: /metrics. The app returned 404 for the old path, so the receiver logged Scrape failed for scrape_pool: otel-collector and the service’s JVM and HTTP metrics vanished from dashboards, even though the app was perfectly healthy.

The fix had two parts. First, metrics_path was corrected to /actuator/prometheus to match the real endpoint, verified with curl -v http://10.0.0.20:8080/actuator/prometheus returning 200. Second, a scrape_timeout: 10s was added because the Actuator endpoint rendered slowly under load and a few scrapes had also been timing out. After reload, up for the target returned to 1 and the metrics reappeared within one scrape interval.

Prevention Best Practices

• Verify every new target with curl -v from the Collector host before trusting the scrape_config.
• Pin metrics_path and scheme explicitly per job instead of relying on defaults that may not match the app.
• Set a realistic scrape_timeout below scrape_interval so slow endpoints fail fast and clearly.
• Store scrape credentials in credentials_file and CA/cert paths in tls_config — never inline secrets in the config.
• Alert on up == 0 per job so a single broken target is caught without waiting for a human to notice a gap.
• Keep scrape configs in version control and review relabel_configs carefully — they silently rewrite target addresses.

Quick Command Reference

# Reproduce the failing scrape by hand
curl -v http://10.0.0.20:8080/metrics

# Watch the receiver's scrape failures
journalctl -u otelcol-contrib -f | grep -i 'Scrape failed'

# Check the target port is reachable
nc -zv 10.0.0.20 8080

# Validate the receiver config after editing scrape_configs
otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml

Conclusion

Scrape failed in the prometheus receiver means the Collector reached a target but could not pull valid metrics from it. The log line hands you everything you need — the scrape_pool, the target URL, and the underlying error — so reproduce that exact scrape with curl, then align the scrape_config’s port, metrics_path, scheme, TLS, and auth with what the target actually serves. Alerting on up == 0 turns a silent missing-metrics gap into an immediate, per-target signal.

Related

• OpenTelemetry Error Guide: ‘duplicate metric with different label sets’
• OpenTelemetry Error Guide: ‘connection refused’ to the Collector
• OpenTelemetry Error Guide: ‘instrument has exceeded the maximum allowed cardinality’