🚨 TLS & Security Error Guide: 'x509: certificate signed by unknown authority'

Overview

x509: certificate signed by unknown authority is the Go TLS stack’s way of saying it built a certificate chain that does not terminate in a CA it trusts. The leaf and intermediates may be perfectly valid and in-date, but the root that anchors them is not present in the client’s trust store. It is the Go equivalent of OpenSSL’s “self signed certificate in certificate chain” / verify code 19 — a trust anchor problem, not an expiry or hostname problem.

You will see it from Docker, kubectl, Helm, Vault, Terraform providers, and any Go program:

x509: certificate signed by unknown authority

Often with the issuer named:

Get "https://registry.internal.example.com/v2/": x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "Example Internal Root CA")

The two common shapes are: (a) a private/internal CA the client has never been told to trust, and (b) a public chain where the server omitted the intermediate, so the client cannot link the leaf to a known root. Both resolve to “no path to a trusted anchor.”

Symptoms

• Go-based tools (docker pull, kubectl, vault, helm) fail with “signed by unknown authority”.
• Browsers may work (they ship a broad root store and fetch intermediates) while these tools fail.
• The endpoint uses an internal/private CA, or a recently rotated public root.
• Adding the CA to one tool’s flag fixes it, proving the system store lacks the root.

echo | openssl s_client -connect registry.internal.example.com:443 -showcerts 2>/dev/null \
  | openssl storeutl -noout -text -certs /dev/stdin 2>/dev/null | grep -E 'Subject:|Issuer:'

        Subject: CN = registry.internal.example.com
        Issuer: CN = Example Internal Issuing CA
        Subject: CN = Example Internal Issuing CA
        Issuer: CN = Example Internal Root CA

The chain ends at Example Internal Root CA — if that root is not trusted, Go reports “unknown authority.”

Common Root Causes

1. Private/internal CA not installed in the system trust store

The endpoint is signed by an internal CA that the host has never been configured to trust.

echo | openssl s_client -connect registry.internal.example.com:443 -showcerts 2>/dev/null \
  | openssl x509 -noout -issuer
sudo ls /usr/local/share/ca-certificates/ /etc/pki/ca-trust/source/anchors/ 2>/dev/null

issuer=CN = Example Internal Root CA

(Empty anchor directory means the internal root was never added.)

2. Server omits the intermediate (incomplete chain to a public root)

The leaf is publicly signed, but only the leaf is served, so Go cannot reach the known root.

echo | openssl s_client -connect registry.internal.example.com:443 -showcerts 2>/dev/null \
  | grep -c 'BEGIN CERTIFICATE'

1

One cert for a public leaf = missing intermediate; Go has no AIA fetch fallback.

3. A container/image without the CA bundle or the internal root

The app runs in a slim image that lacks ca-certificates or was never given the internal root.

docker run --rm registry.internal.example.com/app:latest sh -c 'ls -l /etc/ssl/certs/ca-certificates.crt 2>/dev/null || echo missing'

missing

No trust bundle inside the container means every TLS call is “unknown authority.”

4. Custom CA path not honored by the Go program

Go reads SSL_CERT_FILE/SSL_CERT_DIR on Linux; if the root lives outside those, it is ignored.

env | grep -iE 'SSL_CERT_FILE|SSL_CERT_DIR'
ls -l /etc/ssl/certs/ca-certificates.crt

SSL_CERT_FILE=/opt/legacy/ca.pem

An override pointing at a bundle without the internal root breaks Go specifically.

5. System trust store not refreshed after adding the root

The root file was dropped in the anchor directory but update-ca-certificates was never run to rebuild the bundle.

ls -l /usr/local/share/ca-certificates/internal-root.crt
grep -c 'Example Internal Root CA' /etc/ssl/certs/ca-certificates.crt

-rw-r--r-- 1 root root 1480 Jun 23 13:50 /usr/local/share/ca-certificates/internal-root.crt
0

The source cert exists but is not in the compiled bundle — the rebuild step was skipped.

6. A daemon-specific trust store (Docker registry / Kubernetes)

Docker reads /etc/docker/certs.d/<registry>/ca.crt, separate from the OS store; placing the root only system-wide may not satisfy the daemon.

sudo ls -l /etc/docker/certs.d/registry.internal.example.com:443/ 2>/dev/null || echo "no per-registry CA dir"

no per-registry CA dir

Docker needs the root in its per-registry path, not just the OS bundle.

Diagnostic Workflow

Step 1: Identify the anchoring issuer and whether the chain is complete

echo | openssl s_client -connect <HOST>:443 -showcerts 2>/dev/null \
  | openssl storeutl -noout -text -certs /dev/stdin 2>/dev/null | grep -E 'Subject:|Issuer:'
echo | openssl s_client -connect <HOST>:443 -showcerts 2>/dev/null | grep -c 'BEGIN CERTIFICATE'

A single cert = missing intermediate (fix the server). A complete chain ending in a private root = trust-store problem (fix the client).

Step 2: Check whether that root is in the system store

openssl x509 -in /path/to/expected-root.crt -noout -subject -fingerprint -sha256
grep -c "$(openssl x509 -in /path/to/expected-root.crt -noout -subject_hash)" /dev/null 2>/dev/null
awk -v RS= '/Internal Root CA/' /etc/ssl/certs/ca-certificates.crt | head -1

If the root’s subject does not appear in the bundle, it is not trusted.

Step 3: Confirm Go specifically by testing with an explicit CA

# Most Go tools accept a --cacert/CA flag; for a quick proof use curl which shares the store:
curl --cacert /path/to/internal-root.crt https://<HOST>/ -sS -o /dev/null -w '%{http_code}\n'

A success with --cacert proves the default store is the gap.

Step 4: Install the root into the correct store(s)

# Debian/Ubuntu
sudo cp internal-root.crt /usr/local/share/ca-certificates/internal-root.crt
sudo update-ca-certificates
# RHEL/Fedora
sudo cp internal-root.crt /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract

Step 5: For daemons, add the daemon-specific path and restart

sudo mkdir -p /etc/docker/certs.d/<HOST>:443
sudo cp internal-root.crt /etc/docker/certs.d/<HOST>:443/ca.crt
sudo systemctl restart docker
docker pull <HOST>/app:latest

Example Root Cause Analysis

A new Kubernetes node cannot pull images from the internal Harbor registry; the kubelet logs:

failed to pull image "registry.internal.example.com/team/app:1.4": x509: certificate signed by unknown authority

Other, older nodes pull fine. Inspecting the chain shows it is complete and ends at the company’s private root:

echo | openssl s_client -connect registry.internal.example.com:443 -showcerts 2>/dev/null \
  | openssl storeutl -noout -text -certs /dev/stdin 2>/dev/null | grep -E 'Subject:|Issuer:' | tail -2

        Subject: CN = Example Internal Issuing CA
        Issuer: CN = Example Internal Root CA

So the chain is fine — this is a trust-anchor gap on the new node. Checking the node’s container runtime trust path:

sudo ls /etc/containerd/certs.d/registry.internal.example.com/ 2>/dev/null || echo "missing"
grep -c 'Example Internal Root CA' /etc/ssl/certs/ca-certificates.crt

missing
0

The node image was rebuilt from a newer base that no longer baked in the internal root, and the bootstrap that drops it into the trust store had not run. The older nodes still carried the root from the previous image.

Fix: install the internal root into the node’s system store (and the runtime’s per-registry path) via the node bootstrap, then refresh:

sudo cp internal-root.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
sudo systemctl restart containerd
crictl pull registry.internal.example.com/team/app:1.4

Image is up to date for sha256:...

The node pulls images, and the bootstrap is corrected so future nodes ship the root.

Prevention Best Practices

• Distribute your internal root CA to every host via config management and run update-ca-certificates/update-ca-trust as part of provisioning, not after an outage.
• Bake the internal root into base container images, or mount it, so slim images are not missing the bundle. Verify the root is present in CI before shipping images.
• Fix incomplete public chains on the server (serve fullchain.pem) — Go does not fetch missing intermediates the way browsers do.
• Remember daemon-specific stores: Docker (/etc/docker/certs.d), containerd (/etc/containerd/certs.d), and Java keystores are separate from the OS bundle.
• Track the root’s fingerprint in inventory so a node missing it is caught by a compliance check. See the security hardening guides for a trust-distribution baseline.
• When a fleet rollout breaks pulls or API calls, the free incident assistant can confirm from the chain whether it is a missing-intermediate (server) or missing-root (client) cause.

Quick Command Reference

# Anchoring issuer + chain completeness
echo | openssl s_client -connect <HOST>:443 -showcerts 2>/dev/null \
  | openssl storeutl -noout -text -certs /dev/stdin 2>/dev/null | grep -E 'Subject:|Issuer:'
echo | openssl s_client -connect <HOST>:443 -showcerts 2>/dev/null | grep -c 'BEGIN CERTIFICATE'

# Prove the default store is the gap
curl --cacert /path/to/internal-root.crt https://<HOST>/ -sS -o /dev/null -w '%{http_code}\n'

# Install the root (Debian/Ubuntu)
sudo cp internal-root.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
# Install the root (RHEL/Fedora)
sudo cp internal-root.crt /etc/pki/ca-trust/source/anchors/ && sudo update-ca-trust extract

# Docker per-registry trust
sudo mkdir -p /etc/docker/certs.d/<HOST>:443
sudo cp internal-root.crt /etc/docker/certs.d/<HOST>:443/ca.crt && sudo systemctl restart docker

# Go-specific override check
env | grep -iE 'SSL_CERT_FILE|SSL_CERT_DIR'

Conclusion

x509: certificate signed by unknown authority means Go could not reach a trusted root, and the fix depends on which link is missing:

1. A private/internal CA is not in the system trust store — install it and refresh the bundle.
2. The server omits the intermediate so a public chain cannot reach a known root — serve the full chain.
3. A slim container lacks ca-certificates or the internal root — bake or mount it.
4. A SSL_CERT_FILE/SSL_CERT_DIR override points Go at a bundle without the root.
5. The root file was dropped in but update-ca-certificates/update-ca-trust was never run.
6. A daemon-specific store (Docker/containerd) needs the root in its own path.

Inspect the chain to decide server-side (missing intermediate) vs client-side (missing root), then install the anchor in the correct store and refresh — Go trusts only what is in the compiled bundle.