🚨 Grafana Error Guide: 'x509: certificate signed by unknown authority' — trusting a TLS datasource CA

Overview

Grafana talks to most datasources — Prometheus, Loki, Elasticsearch, InfluxDB — over HTTP, and increasingly over HTTPS. When you point Grafana at an https:// datasource whose certificate is issued by a private or self-signed Certificate Authority, Grafana’s Go net/http client validates the server certificate against the CA bundle inside the Grafana process. If that CA is not trusted, the query fails and the panel shows a red error banner.

Get "https://prometheus:9090/api/v1/query": x509: certificate signed by unknown authority

Two closely related variants come from the same TLS verification path:

Get "https://prometheus.monitoring.svc:9090/api/v1/query": x509: certificate is valid for prometheus-server, not prometheus.monitoring.svc
tls: failed to verify certificate: x509: certificate signed by unknown authority

The first is a SAN (Subject Alternative Name) / hostname mismatch; the second is the generic verification failure. All of them mean Grafana refused to trust the datasource’s TLS certificate.

Symptoms

• A datasource panel shows x509: certificate signed by unknown authority or tls: failed to verify certificate.
• The Save & test button on the datasource config page returns the same error.
• grafana-server logs show t=... lvl=eror ... tsdb.HandleRequest() error wrapping the x509 message.
• The same endpoint works from curl --cacert ca.pem but fails from Grafana.
• LDAP or SMTP over TLS fails with an analogous x509 error during login or alert email delivery.

Common Root Causes

1. Private/self-signed CA not in the Grafana trust store

The datasource cert is signed by an internal CA that the Grafana container’s system bundle (/etc/ssl/certs/ca-certificates.crt) does not include.

# provisioning/datasources/prometheus.yaml — the fix: attach the CA
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: https://prometheus:9090
    jsonData:
      tlsAuth: false
      tlsAuthWithCACert: true
    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkq... (your internal root CA)
        -----END CERTIFICATE-----

lvl=eror msg="Data source proxy request error" error="Get \"https://prometheus:9090/api/v1/query\": x509: certificate signed by unknown authority"

2. Missing intermediate certificate in the chain

The server presents only its leaf cert, not the intermediate that links it to the root. Grafana cannot build a path to a trusted anchor.

# Confirm how many certs the server actually sends (should be leaf + intermediate)
openssl s_client -connect prometheus:9090 -showcerts </dev/null 2>/dev/null \
  | grep -c 'BEGIN CERTIFICATE'

verify error:num=20:unable to get local issuer certificate
verify error:num=21:unable to verify the first certificate

Fix on the server side by concatenating fullchain.pem (leaf + intermediates), or paste the intermediate + root into tlsCACert.

3. SAN / hostname mismatch

The URL you configured does not appear in the certificate’s SAN list. Modern Go ignores the legacy CN field entirely.

openssl s_client -connect prometheus:9090 </dev/null 2>/dev/null \
  | openssl x509 -noout -text | grep -A1 'Subject Alternative Name'

Get "https://prometheus.monitoring.svc:9090/api/v1/query": x509: certificate is valid for prometheus-server, prometheus, not prometheus.monitoring.svc

Point the datasource URL at a name that is in the SAN, or reissue the cert with the correct SANs. As an override you can set jsonData.tlsServerName to a name that is present in the SAN while keeping the real connection host.

4. Expired certificate

An otherwise-trusted cert past its notAfter date also surfaces as a verify failure.

openssl s_client -connect prometheus:9090 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

notBefore=Jan  1 00:00:00 2025 GMT
notAfter=Apr  1 00:00:00 2025 GMT   # in the past → x509: certificate has expired or is not yet valid

Diagnostic Workflow

Step 1 — Reproduce and read the exact error. Open the datasource and click Save & test, then tail the server log:

journalctl -u grafana-server -f --no-pager | grep -i x509
# Kubernetes:
kubectl logs deploy/grafana -n monitoring -f | grep -i x509

Step 2 — Inspect the presented certificate and chain from Grafana’s vantage point. Run openssl from inside the Grafana pod so DNS and network match:

kubectl exec -it deploy/grafana -n monitoring -- \
  sh -c 'openssl s_client -connect prometheus:9090 -showcerts </dev/null'

Step 3 — Decode the leaf for SAN and validity.

openssl s_client -connect prometheus:9090 </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

Step 4 — Verify the chain against your CA file explicitly.

openssl verify -CAfile internal-ca.pem <(openssl s_client -connect prometheus:9090 \
  </dev/null 2>/dev/null | openssl x509)

Step 5 — Apply the trust. Prefer attaching the CA to the datasource (option A). Only fall back to skip-verify (option C) in a lab.

# A) Per-datasource CA (recommended)
jsonData:
  tlsAuthWithCACert: true
secureJsonData:
  tlsCACert: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

# B) Mount the CA into the container trust store (image build or initContainer)
COPY internal-ca.crt /usr/local/share/ca-certificates/internal-ca.crt
RUN update-ca-certificates

# C) Skip verification — INSECURE, disables all cert checks for this datasource
jsonData:
  tlsSkipVerify: true

For SMTP over TLS the equivalent knob lives in grafana.ini:

[smtp]
enabled = true
host = smtp.internal:587
skip_verify = false   ; set true only to bypass an internal CA temporarily

Example Root Cause Analysis

A team migrated Prometheus behind an internal PKI. Grafana panels went red with x509: certificate signed by unknown authority. curl https://prometheus:9090 from a laptop worked because corporate CAs were installed there, masking the problem.

Running openssl s_client from inside the Grafana pod showed verify error:num=20:unable to get local issuer certificate and a single certificate in the chain. Two problems: the internal root CA was absent from the container bundle, and Prometheus served only its leaf. The team rebuilt Prometheus to present fullchain.pem (leaf + intermediate) and added the internal root to the datasource via secureJsonData.tlsCACert with tlsAuthWithCACert: true. After a Save & test, the panel recovered — no tlsSkipVerify needed. They then baked the root CA into the Grafana image with update-ca-certificates so LDAP and SMTP inherited the same trust. See the Grafana troubleshooting guides for related datasource issues.

Prevention Best Practices

• Distribute your internal root CA to every Grafana instance via image build (update-ca-certificates) so all datasources, LDAP and SMTP trust it uniformly.
• Always serve the full chain (leaf + intermediates) from the datasource server; never rely on Grafana holding intermediates.
• Include every hostname Grafana uses (service DNS, FQDN) in the certificate SAN list; CN is ignored.
• Monitor certificate expiry with a Prometheus probe_ssl_earliest_cert_expiry alert.
• Reserve tlsSkipVerify / skip_verify for throwaway labs; it silently defeats MITM protection.

Quick Command Reference

# See the exact error
journalctl -u grafana-server -f | grep -i x509
kubectl logs deploy/grafana -n monitoring -f | grep -i x509

# Inspect the cert Grafana sees (run from the Grafana host/pod)
openssl s_client -connect prometheus:9090 -showcerts </dev/null
openssl s_client -connect prometheus:9090 </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

# Verify chain against your CA
openssl verify -CAfile internal-ca.pem server-leaf.pem

# Add CA to container trust store
cp internal-ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates

Conclusion

The top root causes of x509: certificate signed by unknown authority in Grafana are:

1. The datasource’s private/self-signed CA is not trusted by Grafana’s CA bundle — attach it via tlsCACert / tlsAuthWithCACert or mount it and run update-ca-certificates.
2. A missing intermediate certificate breaks the chain — serve the full chain from the datasource server.
3. A SAN/hostname mismatch (certificate is valid for X, not Y) — fix the URL, reissue with correct SANs, or set tlsServerName.
4. An expired certificate — rotate it and add expiry monitoring.
5. As a last resort in labs only, tlsSkipVerify / skip_verify bypasses validation entirely — never use it in production.