🚨 Prometheus Error Guide: 'server returned HTTP status 401 Unauthorized' Scrape Auth

Exact Error Message

On the Targets page (/targets) the target is DOWN and the Error column shows an HTTP-level rejection — the connection succeeded but the server refused the request:

server returned HTTP status 401 Unauthorized

server returned HTTP status 403 Forbidden

The same string is logged by the scrape manager, often with the scrape URL:

ts=2026-06-27T11:02:55.118Z caller=scrape.go:1351 level=debug component="scrape manager" scrape_pool=kubelet target=https://10.0.0.7:10250/metrics msg="Scrape failed" err="server returned HTTP status 401 Unauthorized"

What the Error Means

Unlike connection refused, this is an application-layer failure: Prometheus reached the target, sent an HTTP request to /metrics, and the server responded with 401 Unauthorized or 403 Forbidden.

• 401 Unauthorized — no credentials were sent, or the credentials are missing/expired. The endpoint requires authentication and the scrape config didn’t supply usable auth.
• 403 Forbidden — credentials were sent and recognized, but the identity is not authorized for /metrics. This is an authorization (RBAC) problem, classically a Kubernetes ServiceAccount lacking the right ClusterRole.

The target shows DOWN and up == 0; every metric it would expose is missing until auth is fixed.

Common Causes

1. Target requires auth, scrape_config has none. The endpoint is protected but the job has no basic_auth, authorization, or bearer_token.
2. Wrong or expired token / password. A rotated API token, changed password, or expired bearer token → 401.
3. bearer_token_file path wrong or unreadable. The file doesn’t exist, has wrong permissions, or holds a stale token.
4. Kubernetes RBAC for the ServiceAccount. The Prometheus SA lacks nodes/metrics, nodes/proxy, or similar on the kubelet/cAdvisor/apiserver → 403.
5. mTLS expected but only a token sent (or vice versa). The server wants a client certificate, not a bearer token.
6. A proxy in front of the target enforces its own auth that Prometheus isn’t satisfying.

How to Reproduce the Error

Scrape an endpoint that requires basic auth without supplying any:

scrape_configs:
  - job_name: "secured-app"
    static_configs:
      - targets: ["10.0.0.8:9000"]
    # no basic_auth / authorization -> 401

Or, in Kubernetes, scrape the kubelet with a ServiceAccount whose ClusterRole omits nodes/metrics:

scrape_configs:
  - job_name: "kubelet"
    scheme: https
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: node

The kubelet authenticates the token (so not 401) but the RBAC check fails → 403 Forbidden.

Diagnostic Commands

List the failing targets and their exact error and URL:

curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.health!="up")
    | [.labels.job, .scrapeUrl, .lastError] | @tsv'

Confirm the auth boundary by curling the target’s /metrics from the Prometheus host without then with credentials:

# Without creds -> reproduce the 401/403
curl -sv https://10.0.0.7:10250/metrics --insecure -o /dev/null

# With a bearer token -> should return metrics (200)
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -s https://10.0.0.7:10250/metrics --insecure \
  -H "Authorization: Bearer ${TOKEN}" | head

Inspect the job’s resolved auth block in the running config:

curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' \
  | grep -A12 'job_name: kubelet'
promtool check config /etc/prometheus/prometheus.yml

For a bearer_token_file, verify it exists, is readable by the Prometheus user, and contains a current token:

sudo -u prometheus cat /etc/prometheus/secrets/token >/dev/null && echo "readable"
ls -l /var/run/secrets/kubernetes.io/serviceaccount/token

For a 403 in Kubernetes, check what the SA is allowed to do:

kubectl auth can-i get nodes/metrics \
  --as=system:serviceaccount:monitoring:prometheus
journalctl -u prometheus --no-pager | grep -iE '401|403|Unauthorized|Forbidden' | tail

Step-by-Step Resolution

Cause: endpoint needs basic auth. Add a basic_auth block (use password_file so the secret isn’t in the YAML).

scrape_configs:
  - job_name: "secured-app"
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/secrets/app_password
    static_configs:
      - targets: ["10.0.0.8:9000"]

Cause: endpoint needs a bearer token / authorization header. Use the authorization stanza.

    authorization:
      type: Bearer
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token

Cause: wrong/expired token. Rotate it, update the file, and reload — the token file is re-read each scrape, so no restart is needed.

curl -X POST http://localhost:9090/-/reload

Cause: Kubernetes 403 (RBAC). Grant the SA the missing verbs/resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources: ["nodes", "nodes/metrics", "nodes/proxy", "services", "endpoints", "pods"]
    verbs: ["get", "list", "watch"]

Bind it to the Prometheus SA with a ClusterRoleBinding, then re-check with kubectl auth can-i.

Cause: mTLS expected. Supply a client cert in tls_config (see the TLS guide) instead of, or in addition to, the token.

After any change validate with promtool, reload, and refresh /targets — the target should return to UP.

Prevention and Best Practices

• Keep credentials out of prometheus.yml: use password_file / credentials_file and mount secrets, so config can live in git safely.
• Alert on token expiry where the issuer exposes it, and rotate via the file (which is hot-reloaded) rather than editing inline YAML.
• For Kubernetes, define the Prometheus ClusterRole/Binding as part of the install (Helm chart or manifest) so RBAC stays correct across upgrades.
• Distinguish 401 from 403 in runbooks: 401 = fix/supply credentials; 403 = fix authorization/RBAC. They have different owners.
• Restrict file permissions on token/password files to the Prometheus user (0640, owned by prometheus).
• Test new secured jobs with a manual curl -H "Authorization: Bearer ..." from the Prometheus host before wiring them into a scrape config.

Related Errors

• Prometheus Error: target DOWN and up == 0 triage hub
• Prometheus Error: connect connection refused scrape DOWN
• Prometheus Error: x509 certificate signed by unknown authority
• Prometheus Error: context deadline exceeded scrape timeout

Frequently Asked Questions

What is the difference between 401 and 403 in a Prometheus scrape? 401 Unauthorized means no valid credentials were presented — Prometheus either sent nothing or an expired/invalid token. 403 Forbidden means the credentials were accepted but the identity isn’t permitted to read /metrics — almost always a Kubernetes RBAC gap. Fix 401 by supplying credentials; fix 403 by granting permissions.

Where do I put the credentials in the scrape config? Use basic_auth (username + password_file) for HTTP basic auth, or the authorization block (type: Bearer, credentials_file) for token-based auth. Prefer the _file variants so secrets stay out of the YAML and are hot-reloaded.

My kubelet scrape returns 403 even though the token is valid — why? The token authenticates the ServiceAccount, but the SA’s ClusterRole lacks nodes/metrics (or nodes/proxy). Run kubectl auth can-i get nodes/metrics --as=system:serviceaccount:<ns>:<sa>; if it says no, add those resources to the ClusterRole and rebind.

Do I need to restart Prometheus after rotating a token? No. credentials_file / password_file are read on each scrape, so updating the file (and sending a POST /-/reload if you also changed the config) is enough — no full restart required.

Could a 401 actually be a TLS problem? No — a 401 is a completed HTTP response, so the TLS handshake already succeeded. If TLS were broken you’d see x509: ... errors instead. See the TLS guide for those.