🚨 Prometheus Error Guide: 'x509: certificate has expired or is not yet valid' Scrape Failure

Exact Error Message

This is a TLS verification failure on the scrape connection. Prometheus completed the TCP handshake but rejected the target’s certificate because it is outside its validity window. It appears in the Prometheus log as a Scrape failed warning:

level=warn ts=2026-06-27T09:14:00.231Z caller=scrape.go:1654 component="scrape manager" scrape_pool=node msg="Scrape failed" err="Get \"https://10.0.4.7:9100/metrics\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-06-27T09:14:00Z is after 2026-06-20T00:00:00Z"

The same target shows up as DOWN on the Status → Targets page, with the last error repeated verbatim:

Endpoint                          State   Labels              Last Scrape   Error
https://10.0.4.7:9100/metrics     DOWN    instance="node1"    4.512s ago    Get "https://10.0.4.7:9100/metrics": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-06-27T09:14:00Z is after 2026-06-20T00:00:00Z

The “not yet valid” variant is the mirror image — the certificate’s NotBefore is in the future relative to whatever clock is doing the comparison:

err="Get \"https://10.0.4.7:9100/metrics\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-06-27T09:14:00Z is before 2026-06-28T00:00:00Z"

Note the tail of the string: is after <NotAfter> means expired; is before <NotBefore> means not-yet-valid. That single word tells you which end of the window you crossed.

What the Error Means

Every X.509 certificate carries a validity window: NotBefore (when it starts being valid) and NotAfter (when it expires). When Prometheus scrapes over HTTPS, Go’s TLS stack checks the target’s presented certificate against the current system time on the Prometheus host. If now is past NotAfter, the cert is expired. If now is before NotBefore, it is not yet valid. Either way verification fails, the scrape is aborted before any metrics are read, and the target flips to DOWN.

This is distinct from x509: certificate signed by unknown authority, which is about trust — Prometheus cannot build a chain to a CA it trusts. Here the chain may be perfectly trusted; the cert is simply outside its time window. It is also distinct from a connection refused failure, where no TLS handshake happens at all because nothing is listening.

Critically, “expired or is not yet valid” is time-relative. A perfectly good certificate can trip this error purely because a clock is wrong — the target’s, or Prometheus’s. So the validity window in the log message is only half the story; the other half is the current time the error prints next to it.

Common Causes

• The target’s leaf certificate is genuinely past NotAfter. Someone forgot to rotate it, or a manual rotation lapsed. The classic ops own-goal.
• cert-manager renewal failed. The Certificate object’s renewal didn’t complete (failed ACME challenge, rate limit, RBAC, stuck order), so the old secret is still mounted on the exporter pod past its expiry.
• Expired Let’s Encrypt certificate. The 90-day cert wasn’t renewed by certbot/the ACME client, often because the renew timer was disabled or DNS/HTTP validation broke.
• Clock skew makes a valid cert look expired or not-yet-valid. If the Prometheus host clock is far ahead, a current cert appears expired; if it is behind, a freshly issued cert appears not-yet-valid. A target whose own clock is wrong can also present a cert with a future NotBefore.
• An expired intermediate certificate. The leaf is fine, but an intermediate in the chain expired, and the verification message reports the whole chain as invalid.
• A stale CA/cert bundle pinned in tls_config. A ca_file (or a pinned client cert) baked at deploy time no longer matches the rotated chain on the target.
• A container image baked with an old certificate. The cert was copied into the image at build time and never refreshed; the running container serves a cert that expired weeks ago.

How to Reproduce the Error

Issue a deliberately short-lived self-signed cert, serve it, and scrape it after it expires. Generate a cert valid for a single day:

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/node.key -out /tmp/node.crt \
  -days 1 -subj "/CN=node1" -addext "subjectAltName=IP:127.0.0.1"

Point a node_exporter (or any HTTPS endpoint) at it, add the target to prometheus.yml under an HTTPS scrape config, and let a day pass — or, to force the “not yet valid” case immediately, set the target host’s clock a week ahead before issuing the cert so its NotBefore lands in Prometheus’s future. Then watch the scrape fail:

journalctl -u prometheus -f | grep -i 'x509'

msg="Scrape failed" err="... x509: certificate has expired or is not yet valid: current time 2026-06-27T09:14:00Z is after 2026-06-20T00:00:00Z"

Diagnostic Commands

All of these are read-only — they inspect state, they don’t change it.

Pull the target cert’s validity window, subject, and issuer directly off the wire:

openssl s_client -connect 10.0.4.7:9100 -servername node1 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

notBefore=May 22 00:00:00 2026 GMT
notAfter=Jun 20 00:00:00 2026 GMT
subject=CN=node1
issuer=CN=internal-ca

Just the expiry, for a quick yes/no:

echo | openssl s_client -connect 10.0.4.7:9100 2>/dev/null | openssl x509 -noout -enddate

curl reports the same expiry in its verbose TLS output and fails the same way Prometheus does:

curl -vI https://10.0.4.7:9100/metrics

* SSL certificate problem: certificate has expired

Confirm the clocks before you trust the cert’s verdict — a wrong clock fakes this error:

date -u
chronyc tracking | grep -E 'System time|Last offset'

System time     : 0.000312 seconds slow of NTP time

And ask the Prometheus API directly which targets are down and why:

curl -s http://localhost:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.health=="down") | {scrapeUrl, lastError}'

{ "scrapeUrl": "https://10.0.4.7:9100/metrics",
  "lastError": "... x509: certificate has expired or is not yet valid: current time ... is after 2026-06-20T00:00:00Z" }

Step-by-Step Resolution

1. Confirm the clock first — it’s the cheap check. Run date -u and chronyc tracking on both the Prometheus host and the target. If either clock is off by hours or days, the cert may be fine and you’re chasing a phantom. Fix NTP and re-evaluate:

sudo chronyc makestep
systemctl status chronyd

A “not yet valid” error with sane cert dates is almost always a clock that’s behind.

2. Inspect the actual cert expiry. Use the openssl s_client … -dates command above against the real target. Compare notAfter to date -u. This tells you definitively whether the leaf is expired, the window is in the future, or an intermediate is the culprit (check the full chain with openssl s_client -showcerts).

3. Renew or rotate the target’s certificate. This is the real fix:

• cert-manager: check the object and force renewal.

  kubectl get certificate -n monitoring node1-tls -o wide
  kubectl describe certificate -n monitoring node1-tls   # read the Conditions for the renewal failure
  kubectl cmctl renew node1-tls -n monitoring

• Let’s Encrypt: renew and verify the timer is enabled.

  sudo certbot renew --cert-name node1.example.com
  systemctl status certbot.timer

• Internal CA: re-issue the leaf from your CA and install the new cert + key on the target.

4. Reload the target so it serves the new cert. Most exporters need a restart or reload to pick up a rotated cert/key on disk (cert-manager-mounted secrets update in the pod, but the process may still need a SIGHUP/restart). Once the target presents a cert inside its validity window, the next scrape succeeds and the target returns to UP on its own — no Prometheus restart required.

5. Update tls_config if an intermediate or CA changed. If the rotation introduced a new intermediate or CA, point Prometheus’s ca_file at the current bundle and reload Prometheus:

scrape_configs:
  - job_name: node
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/ca/internal-ca-bundle.pem
      server_name: node1

curl -X POST http://localhost:9090/-/reload

Do not “fix” this by disabling verification. It is tempting to silence the error with:

tls_config:
  insecure_skip_verify: true   # security anti-pattern — avoid

This turns off certificate validation entirely, leaving the scrape open to interception and hiding the next expiry from you. The correct fix is to rotate the cert. Treat insecure_skip_verify only as a deliberate, documented, temporary escape hatch (e.g. a lab with a known self-signed endpoint) — never as the resolution to an expired cert in production.

Prevention and Best Practices

• Automate renewal and prove it works. Use cert-manager or an ACME client with a healthy renew timer, and alert when a Certificate enters a non-Ready state rather than discovering it at expiry.
• Alert on days-to-expiry, not on the outage. The blackbox_exporter exposes probe_ssl_earliest_cert_expiry; alert when (probe_ssl_earliest_cert_expiry - time()) / 86400 drops below ~14 days so you rotate before Prometheus ever logs x509.
• Run NTP/chrony everywhere and alert on offsets over ~1 s. A surprising share of “expired” errors are wrong clocks, not wrong certs.
• Never bake certs into container images. Mount them from a secret that rotates; an image-baked cert expires silently and survives every redeploy.
• Keep ca_file bundles in sync with rotation. When you rotate an internal CA or intermediate, update the bundle Prometheus pins and reload — or the trust check breaks next.

Related Errors

• x509: certificate signed by unknown authority — a trust failure, not a time failure. The cert window is fine but Prometheus can’t chain it to a trusted CA; the fix is ca_file, not rotation.
• connection refused — no TLS handshake happens at all because nothing is listening on the target port. Different layer, different fix.
• tls: failed to verify certificate: x509: certificate is valid for X, not Y — a SAN/hostname mismatch; set server_name in tls_config to a name the cert actually covers.

Frequently Asked Questions

Is this the same as x509: certificate signed by unknown authority? No. “Unknown authority” is a trust problem — Prometheus doesn’t trust the signing CA. “Expired or is not yet valid” is a time problem — the cert is outside its NotBefore/NotAfter window (or a clock makes it appear so). The fixes differ: trust the CA via ca_file versus rotate the cert.

The cert looks valid in openssl but Prometheus still rejects it. Why? Check the clock on the Prometheus host with date -u. Go compares the cert window against Prometheus’s system time; if that clock is far ahead, a valid cert reads as expired, and if it’s behind, a new cert reads as not-yet-valid.

Can I just set insecure_skip_verify: true? You can, but you shouldn’t. It disables certificate validation entirely, exposing the scrape to interception and masking the next expiry. Use it only as a documented, temporary escape hatch for known-self-signed lab endpoints. Rotate the cert for any real fix.

Why did only one target break when they all use the same CA? That target’s leaf certificate hit its own NotAfter (or its renewal failed), independent of the CA. Each leaf has its own validity window — inspect that specific endpoint with openssl s_client -connect <host:port> -dates.

How do I get alerted before this happens again? Probe the endpoint with blackbox_exporter and alert on probe_ssl_earliest_cert_expiry approaching time(), plus alert on cert-manager Certificate objects that aren’t Ready. Both catch the expiry days before a scrape ever fails.