🚨 Prometheus Error Guide: 'x509: certificate signed by unknown authority' Scrape TLS

Exact Error Message

On the Targets page (/targets) an HTTPS target is DOWN and the Error column shows an x509 verification failure. The three you will paste into Google:

Get "https://10.0.0.7:10250/metrics": x509: certificate signed by unknown authority

Get "https://node1.internal:9100/metrics": x509: certificate is valid for node1.local, not node1.internal

Get "https://10.0.0.7:10250/metrics": x509: certificate has expired or is not yet valid: current time 2026-06-27T11:40:02Z is after 2026-06-01T00:00:00Z

The scrape manager logs the same:

ts=2026-06-27T11:40:02.551Z caller=scrape.go:1351 level=debug component="scrape manager" scrape_pool=kubelet target=https://10.0.0.7:10250/metrics msg="Scrape failed" err="Get \"https://10.0.0.7:10250/metrics\": x509: certificate signed by unknown authority"

What the Error Means

The TCP connection and the start of the TLS handshake succeeded — the target presented a certificate — but Prometheus refused to trust it. Verification can fail three ways:

• signed by unknown authority — the cert chain’s root/intermediate CA is not in Prometheus’s trust store and no ca_file was provided. Common with self-signed certs and private/internal CAs.
• certificate is valid for X, not Y — the cert’s SAN (Subject Alternative Names) doesn’t include the name/IP Prometheus connected to (a hostname/SAN mismatch).
• certificate has expired — the cert’s notAfter is in the past (or notBefore is in the future).

In every case the target shows DOWN with up == 0 and no metrics, even though the network path and the server are healthy.

Common Causes

1. Private CA not trusted. The target uses an internal CA whose root isn’t in the system trust store and tls_config.ca_file is unset → unknown authority.
2. Self-signed certificate with no CA to trust at all.
3. SAN / hostname mismatch. Prometheus connects by IP (10.0.0.7) but the cert only lists a DNS name, or vice versa → “valid for X, not Y”.
4. Expired certificate that was never renewed/rotated.
5. insecure_skip_verify misuse — either it’s missing where intentionally needed, or it’s silently masking a real cert problem.
6. Kubelet serving cert is self-signed (default) and Prometheus is configured to verify it.

How to Reproduce the Error

Scrape an HTTPS endpoint whose cert is signed by a CA Prometheus doesn’t trust, with no tls_config:

scrape_configs:
  - job_name: "kubelet"
    scheme: https
    static_configs:
      - targets: ["10.0.0.7:10250"]
    # no tls_config -> uses system trust store -> unknown authority for private CA

To reproduce the SAN mismatch, target the cert by an address not in its SAN list — connecting to 10.0.0.7 when the cert only lists node1.internal yields x509: certificate is valid for node1.internal, not 10.0.0.7.

Diagnostic Commands

List the failing TLS targets with their exact error:

curl -s http://localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.health!="up")
    | [.labels.job, .scrapeUrl, .lastError] | @tsv'

Inspect the certificate the target actually serves — read its issuer, SANs, and validity dates:

echo | openssl s_client -connect 10.0.0.7:10250 -servername node1.internal 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates -ext subjectAltName

issuer=CN = internal-ca
subject=CN = node1.internal
notBefore=Apr  1 00:00:00 2026 GMT
notAfter=Jun  1 00:00:00 2026 GMT        # <- expired
X509v3 Subject Alternative Name:
    DNS:node1.internal                    # <- no IP SAN

Reproduce Prometheus’s verification from the Prometheus host with and without the CA:

# Fails the same way Prometheus does
curl -v https://10.0.0.7:10250/metrics -o /dev/null

# Succeeds if the CA file is the missing piece
curl -v --cacert /etc/prometheus/certs/internal-ca.pem \
  https://node1.internal:10250/metrics -o /dev/null

Check the job’s resolved tls_config and validate the config:

curl -s http://localhost:9090/api/v1/status/config | jq -r '.data.yaml' | grep -A14 'job_name: kubelet'
promtool check config /etc/prometheus/prometheus.yml
journalctl -u prometheus --no-pager | grep -i x509 | tail

Step-by-Step Resolution

Cause: private/self-signed CA not trusted. Provide the CA bundle in tls_config.ca_file.

scrape_configs:
  - job_name: "kubelet"
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/certs/internal-ca.pem
    kubernetes_sd_configs:
      - role: node

Cause: SAN / hostname mismatch. Either connect by a name that’s in the SAN, or pin the expected name with server_name so verification uses it.

    tls_config:
      ca_file: /etc/prometheus/certs/internal-ca.pem
      server_name: node1.internal   # verify against this SAN, not the dialed IP

A relabel rule can rewrite __address__ to use the DNS name the cert expects, instead of a raw pod/node IP.

Cause: expired certificate. Renew/rotate the cert on the target (or trigger your cert-manager / PKI to reissue). No Prometheus change is needed once the new cert is in place.

Cause: kubelet self-signed serving cert. The proper fix is to enable kubelet serving-cert rotation signed by the cluster CA and point ca_file at that CA. As a last resort only:

    tls_config:
      insecure_skip_verify: true   # disables verification -> no MITM protection; avoid in prod

insecure_skip_verify: true makes the target UP by skipping validation entirely — it hides expired/mismatched/untrusted certs and removes protection against man-in-the-middle. Use a real ca_file instead.

After editing, validate with promtool, reload, and confirm /targets returns to UP:

promtool check config /etc/prometheus/prometheus.yml
curl -X POST http://localhost:9090/-/reload

Prevention and Best Practices

• Distribute your internal CA bundle to Prometheus via ca_file (or add it to the system trust store) rather than reaching for insecure_skip_verify.
• Connect to targets by a name present in the cert SAN, or set server_name explicitly, so IP-based service discovery doesn’t trip the “valid for X, not Y” check.
• Automate certificate issuance and rotation (cert-manager, Vault, internal PKI) so certs renew before notAfter.
• Alert on upcoming expiry with the blackbox exporter’s probe_ssl_earliest_cert_expiry (e.g. fire when under 14 days) for both targets and Prometheus’s own endpoints.
• Treat insecure_skip_verify: true as a temporary diagnostic toggle, never a steady state — it silently masks expired and forged certificates.
• Keep kubelet serving certs signed by the cluster CA (rotation enabled) so you can verify them with a single ca_file.

Related Errors

• Prometheus Error: target DOWN and up == 0 triage hub
• Prometheus Error: 401 Unauthorized / 403 Forbidden on scrape
• Prometheus Error: connect connection refused scrape DOWN
• Prometheus Error: context deadline exceeded scrape timeout

Frequently Asked Questions

What does “x509: certificate signed by unknown authority” mean in a Prometheus scrape? The target served a certificate whose signing CA isn’t in Prometheus’s trust store, and no tls_config.ca_file was provided. This is normal for private/internal CAs and self-signed certs. The fix is to point ca_file at the CA that issued the target’s cert.

How do I fix “certificate is valid for X, not Y”? The cert’s SAN list doesn’t include the address Prometheus dialed. Either connect using a name that’s in the SAN (often by relabeling __address__ to the DNS name) or set tls_config.server_name to the SAN so verification matches. Connecting by raw IP fails unless the cert has an IP SAN.

Is it safe to set insecure_skip_verify: true? Only as a short-term diagnostic. It disables all certificate validation, so the target will appear UP even with an expired, mismatched, or attacker-supplied cert, and it removes MITM protection. Always replace it with a correct ca_file (and server_name if needed).

Why does scraping the kubelet on :10250 always fail x509 verification? By default the kubelet serves a self-signed certificate. Either enable kubelet serving-cert rotation signed by the cluster CA and verify with that CA’s ca_file, or (less ideally) skip verification. The Prometheus community Helm charts configure this for you.

My cert was renewed but Prometheus still shows expired — what now? Confirm the target is actually serving the new cert with openssl s_client ... | openssl x509 -dates. If the old dates still appear, the service wasn’t restarted/reloaded after the cert was replaced. Prometheus re-reads the served cert each scrape, so once the target presents the new one the error clears.