🚨 Telegraf Error Guide: '[inputs.prometheus] connection refused' — Fix Scrape Target Errors

Overview

The prometheus input scrapes a /metrics endpoint and converts the exposition format into Telegraf metrics. When the target exporter is not listening at the configured URL, Telegraf reports the failed HTTP request each interval:

2026-07-12T12:00:00Z E! [inputs.prometheus] Error in plugin: error making HTTP request to "http://localhost:9100/metrics": connection refused

A related variant appears when the target is reachable but rejects the TLS handshake or the request is unauthorized:

E! [inputs.prometheus] Error in plugin: error making HTTP request to "https://10.0.0.20:9100/metrics": x509: certificate signed by unknown authority

This is the Prometheus input (Telegraf acting as a scraper), not outputs.prometheus_client. No metrics from the target appear until the endpoint is reachable and trusted.

Symptoms

• Metrics from a specific exporter (node_exporter, cadvisor, an app /metrics) are missing while other inputs work.
• journalctl -u telegraf repeats connection refused or no such host for the same URL each interval.
• curl http://localhost:9100/metrics fails from the Telegraf host, or succeeds only from a different host.
• In Kubernetes, only some discovered pods fail — usually those without the scrape annotations or with the wrong port.
• The error changed from connection refused to 401 Unauthorized or an x509 message after enabling auth/TLS on the exporter.

Common Root Causes

• Exporter not running — the node_exporter/app process is down or crashed, so nothing listens on the port.
• Wrong port or path — scraping :9100 when the exporter listens on :9256, or /metrics when the app serves /actuator/prometheus.
• Binding to localhost only — the exporter binds 127.0.0.1, so a remote Telegraf gets connection refused while a local scrape works.
• TLS scheme mismatch — using http:// against an HTTPS endpoint (or vice versa), or an untrusted CA producing x509: certificate signed by unknown authority.
• Missing/incorrect bearer token — the endpoint requires Authorization: Bearer ... and returns 401 without it.
• Kubernetes discovery misconfigured — wrong monitor_kubernetes_pods scope, missing prometheus.io/scrape annotations, or RBAC that blocks pod listing.
• Firewall / NetworkPolicy — a host firewall or Kubernetes NetworkPolicy drops the connection to the metrics port.

Diagnostic Workflow

First reproduce the scrape outside Telegraf from the exact host running the agent. If curl fails the same way, the problem is the target or network, not Telegraf:

# Is anything listening on the port locally?
ss -ltnp | grep 9100

# Reproduce the scrape from the Telegraf host
curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:9100/metrics
curl -sS http://10.0.0.20:9100/metrics | head

Run only the prometheus input with debug to confirm the parsed target and status:

telegraf --config /etc/telegraf/telegraf.conf --test --input-filter prometheus --debug

A correct static-target config pins the scheme and, when needed, TLS trust and a token:

[[inputs.prometheus]]
  urls = ["http://10.0.0.20:9100/metrics"]
  metric_version = 2
  response_timeout = "5s"

  # HTTPS target with a private CA:
  # urls = ["https://10.0.0.20:9100/metrics"]
  # tls_ca = "/etc/telegraf/ca.pem"
  # insecure_skip_verify = false

  # Bearer-token-protected endpoint:
  # bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"

For Kubernetes pod discovery, scope it correctly and rely on scrape annotations instead of hard-coded URLs:

[[inputs.prometheus]]
  monitor_kubernetes_pods = true
  kubernetes_namespace = "monitoring"          # or omit to watch all namespaces
  monitor_kubernetes_pods_scheme = "http"
  # Pods must carry: prometheus.io/scrape: "true", prometheus.io/port: "9100"

If the exporter binds to localhost, fix it at the source (for node_exporter, --web.listen-address=0.0.0.0:9100) rather than working around it in Telegraf.

Example Root Cause Analysis

After a node_exporter upgrade rolled out by config management, a monitoring host started logging [inputs.prometheus] ... error making HTTP request to "http://10.0.0.20:9100/metrics": connection refused. ping 10.0.0.20 worked and the exporter process was clearly running (systemctl status node_exporter was active). Running ss -ltnp | grep 9100 on the target revealed the listener was 127.0.0.1:9100, not 0.0.0.0:9100 — the new package default had changed the bind address to localhost.

Because Telegraf scraped from a separate monitoring host, every request was refused at the target’s loopback boundary. Setting --web.listen-address=0.0.0.0:9100 on node_exporter and restarting it made the scrape succeed immediately. The lesson: connection refused from a running exporter almost always means it is listening on the wrong interface or port — check the listener with ss on the target, not just whether the process is up.

Prevention Best Practices

• Reproduce every scrape target with curl from the Telegraf host before trusting the config; automate it as a post-deploy check.
• Pin the correct scheme (http/https) and provide tls_ca for private CAs instead of setting insecure_skip_verify = true.
• Confirm exporters bind a routable interface (0.0.0.0), not just loopback, when Telegraf scrapes remotely.
• In Kubernetes, standardize prometheus.io/scrape/prometheus.io/port annotations and grant the agent’s service account list/watch RBAC on pods.
• Set a realistic response_timeout so a slow target fails fast rather than stalling the input.
• Alert on the prometheus input’s own gather errors so a downed exporter is caught before dashboards go blank.

Quick Command Reference

# Check the target's listener (run on the target host)
ss -ltnp | grep 9100

# Reproduce the scrape from the Telegraf host
curl -sS -o /dev/null -w '%{http_code}\n' http://10.0.0.20:9100/metrics

# Run only the prometheus input with debug
telegraf --config /etc/telegraf/telegraf.conf --test --input-filter prometheus --debug

# Tail scrape errors live
journalctl -u telegraf -f | grep -i prometheus

Related Guides

• Telegraf HTTP connection refused
• Telegraf x509 certificate signed by unknown authority
• Telegraf prometheus_client address already in use

More fixes in the Telegraf guides.

Conclusion

[inputs.prometheus] ... connection refused means Telegraf’s scraper could not reach the exporter at the configured URL — usually because the target is down, listening on the wrong port or interface, or behind a TLS/auth requirement the input does not satisfy. Reproduce with curl from the Telegraf host, check the listener with ss on the target, and pin the correct scheme, tls_ca, or bearer_token. For Kubernetes, fix pod discovery scope and annotations, and the target’s metrics will scrape reliably.