🚨 VictoriaMetrics Error: 'x509: certificate signed by unknown authority' — Cause, Fix, and Troubleshooting Guide

What this error means

When vmagent ships samples to a remote_write endpoint over HTTPS, it validates the server’s TLS certificate against a trusted certificate authority. If that chain cannot be verified, the send fails during the TLS handshake and the block is retried but never accepted:

cannot send a block to remote storage "https://metrics.example.com/api/v1/write": Post ... x509: certificate signed by unknown authority

This is a client-side trust failure: vmagent reached the server, but the certificate the server presented is not anchored to a CA vmagent trusts. It is distinct from a refused connection (nothing listening) and from an HTTP 429 (accepted then rate-limited). The fix is almost always to give vmagent the right CA, not to disable verification.

Symptoms at the client

• vmagent logs x509: certificate signed by unknown authority repeatedly for one remote_write URL.
• The remote storage receives no data while local scraping continues, so vmagent’s on-disk queue grows.
• vmagent_remotewrite_errors_total climbs for the affected -remoteWrite.url.
• The same endpoint works from a browser or curl that trusts a corporate CA vmagent does not have.

Trust and cipher causes

• The server’s CA is not trusted — vmagent runs in a minimal image with no corporate/internal CA bundle installed.
• A self-signed certificate — the endpoint presents a cert with no external chain of trust.
• Hostname/SAN mismatch — the certificate is otherwise valid but its SAN does not match the host in -remoteWrite.url (this may also surface as a name-mismatch x509 error).
• An expired certificate — the chain was valid but the leaf or an intermediate has passed its notAfter date.

Inspecting the TLS handshake

Inspect the certificate the endpoint actually serves, including its issuer and SANs:

echo | openssl s_client -connect metrics.example.com:443 -servername metrics.example.com 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates -ext subjectAltName

Test whether the chain verifies against a specific CA file the way vmagent would:

curl -v --cacert /etc/vmagent/ca.crt 'https://metrics.example.com/api/v1/write'

Check what CA and server name vmagent is currently configured with:

ps aux | grep vmagent | grep -oE '\-remoteWrite.tls[^ ]*'

The fix

1. Point vmagent at the CA with -remoteWrite.tlsCAFile. Supply the CA (or full chain) that issued the server certificate:

./vmagent -remoteWrite.url='https://metrics.example.com/api/v1/write' \
  -remoteWrite.tlsCAFile='/etc/vmagent/ca.crt'

For multiple remote_write targets, the tls* flags are positional — order them to match each -remoteWrite.url.

2. Set -remoteWrite.tlsServerName to match the certificate SAN. When you connect via an IP or a name that differs from the cert’s SAN, tell vmagent which name to verify:

./vmagent -remoteWrite.url='https://10.0.0.20/api/v1/write' \
  -remoteWrite.tlsCAFile='/etc/vmagent/ca.crt' \
  -remoteWrite.tlsServerName='metrics.example.com'

3. Renew an expired cert. If -dates shows the certificate has expired, reissue it on the server side; no client flag can make vmagent trust an expired chain.

4. As a last resort in trusted networks, use -remoteWrite.tlsInsecureSkipVerify=true. This disables verification entirely and should only ever be used on a private, controlled network while you fix the real trust problem:

./vmagent -remoteWrite.url='https://metrics.example.com/api/v1/write' \
  -remoteWrite.tlsInsecureSkipVerify=true

Certificate lifecycle

• tlsInsecureSkipVerify=true defeats the purpose of TLS and exposes you to interception; treat it as temporary, never a fix committed to production.
• The tls* flags are per-remoteWrite and positional; mixing up their order silently applies the wrong CA to the wrong endpoint.
• Missing an intermediate certificate looks identical to an untrusted root — provide the full chain in the CA file if verification still fails.
• A cert that verifies today can expire later; monitor certificate expiry so remote_write does not break unattended.

Related TLS errors

• VictoriaMetrics Error Guide: remote write connection refused
• VictoriaMetrics Error Guide: remote write 429
• VictoriaMetrics Error Guide: connection reset by peer