🚨 Loki Error Guide: 'x509: certificate signed by unknown authority' — Trust the CA in the Client tls_config

Overview

This Go TLS error appears when a client cannot verify the certificate presented by the server it is connecting to. In a Loki deployment it shows up when Promtail or Alloy pushes logs to Loki over TLS, or when Loki itself talks to object storage or a gateway that uses a private CA:

x509: certificate signed by unknown authority

It is commonly wrapped in a client log line, for example:

level=error msg="final error sending batch" error="Post \"https://loki-gateway/loki/api/v1/push\": x509: certificate signed by unknown authority"

The message means the server’s certificate is valid, but the chain leading to it ends at a Certificate Authority the client does not trust. That happens when the cert is signed by a private/internal CA, when the presented chain is missing an intermediate, when the cert is self-signed, or when the client’s ca_file points at the wrong bundle. The fix is to give the client the CA (and any intermediates) it needs to complete the chain — not to disable verification.

Symptoms

• Promtail/Alloy logs x509: certificate signed by unknown authority and stops delivering batches to Loki.
• A browser or curl --cacert with the right CA succeeds, while the client with the system trust store fails.
• Loki logs the same x509 error when reaching object storage (S3/GCS/MinIO) fronted by a private CA.
• The error started after rotating to an internal CA or moving behind a TLS-terminating gateway.
• openssl s_client shows a verify error unable to get local issuer certificate.

Common Root Causes

• Private/internal CA not in the client trust store — an internal PKI signed the server cert and the client has never been given that root.
• Missing intermediate chain — the server presents only its leaf certificate, so the client cannot build a path to the root.
• Self-signed certificate — no CA at all, so nothing verifies unless the client explicitly trusts that exact cert.
• Wrong ca_file path — tls_config.ca_file points at a missing, empty, or unrelated file.
• CA bundle not mounted into the pod — the file exists on a node or in a secret but was never mounted into the client container.

How to diagnose

1. Reproduce the failure directly against the endpoint with the client’s view of trust:

   curl -v https://loki-gateway/loki/api/v1/push
   # look for: SSL certificate problem: unable to get local issuer certificate

2. Inspect the chain the server actually presents — a missing intermediate shows here:

   openssl s_client -connect loki-gateway:443 -showcerts </dev/null
   # count the certificates returned; a lone leaf means the chain is incomplete

3. Verify the leaf against your CA bundle to confirm the bundle is the right one:

   openssl verify -CAfile /etc/ssl/certs/internal-ca.pem server-leaf.pem

4. Confirm the CA file is present inside the client pod, not just on the host:

   kubectl exec deploy/promtail -- ls -l /etc/ssl/certs/internal-ca.pem

5. Point the client config at that CA and confirm the path matches what is mounted:

   clients:
     - url: https://loki-gateway/loki/api/v1/push
       tls_config:
         ca_file: /etc/ssl/certs/internal-ca.pem

Fixes

Add the CA to the client tls_config.ca_file so the client can build the chain to your internal root. For Promtail:

clients:
  - url: https://loki-gateway/loki/api/v1/push
    tls_config:
      ca_file:   /etc/ssl/certs/internal-ca.pem
      cert_file: /etc/ssl/certs/client.pem   # only if mTLS is required
      key_file:  /etc/ssl/private/client.key

For Alloy, set the CA on the client tls_config block:

loki.write "default" {
  endpoint {
    url = "https://loki-gateway/loki/api/v1/push"
    tls_config {
      ca_file = "/etc/ssl/certs/internal-ca.pem"
    }
  }
}

Mount the CA bundle into the pod so the ca_file path actually resolves inside the container:

# Deployment excerpt: mount the CA from a Secret
volumes:
  - name: internal-ca
    secret:
      secretName: internal-ca-bundle
volumeMounts:
  - name: internal-ca
    mountPath: /etc/ssl/certs/internal-ca.pem
    subPath: ca.pem
    readOnly: true

Include the full chain when the server omits intermediates — concatenate leaf, intermediate(s), and root so any client can build a path, and point the server at the full bundle:

# build a fullchain the server presents (leaf -> intermediate -> root)
cat server-leaf.pem intermediate.pem internal-ca.pem > fullchain.pem

Set the CA on Loki’s storage client when Loki is the one that cannot verify object storage behind a private CA:

storage_config:
  aws:
    s3: https://minio.internal:9000/loki
    http_config:
      ca_file: /etc/ssl/certs/internal-ca.pem

Use insecure_skip_verify only for a throwaway test, never in production — it disables verification entirely and invites man-in-the-middle attacks. Prefer fixing trust; if you must confirm the endpoint is otherwise reachable:

tls_config:
  insecure_skip_verify: true   # TEST ONLY — remove before production

What to watch out for

• insecure_skip_verify: true makes the error disappear by turning off the very check that protects you; treat any config that ships with it as unfinished.
• A missing intermediate fails only for clients that lack it cached — it can pass on one machine and fail on another, which looks intermittent but is not.
• Updating the node/system trust store does not help a container that ships its own trust store; you must mount the CA into the pod.
• Rotating the internal CA breaks every client that pins only the old root — roll out the new CA to clients before switching the server cert.
• The ca_file must be readable by the process user; a mounted-but-unreadable file fails the same way as a missing one.

Related

• Loki Error Guide: ‘no org id’ — the other push-path failure to rule out once TLS trust is fixed.
• Loki Error Guide: ‘failed to flush chunks’ — what an ingester logs when it cannot reach object storage, sometimes for the same TLS reason.
• Loki Error Guide: ‘context deadline exceeded’ — the timeout that can mask a slow or failing TLS handshake.