🚨 VictoriaMetrics Error: 'x509: certificate signed by unknown authority' — Cause, Fix, and Troubleshooting Guide

Overview

When vmagent ships samples to a remote_write endpoint over HTTPS, it validates the server’s TLS certificate against a trusted certificate authority. If that chain cannot be verified, the send fails during the TLS handshake and the block is retried but never accepted:

cannot send a block to remote storage "https://metrics.example.com/api/v1/write": Post ... x509: certificate signed by unknown authority

This is a client-side trust failure: vmagent reached the server, but the certificate the server presented is not anchored to a CA vmagent trusts. It is distinct from a refused connection (nothing listening) and from an HTTP 429 (accepted then rate-limited). The fix is almost always to give vmagent the right CA, not to disable verification.

Symptoms

• vmagent logs x509: certificate signed by unknown authority repeatedly for one remote_write URL.
• The remote storage receives no data while local scraping continues, so vmagent’s on-disk queue grows.
• vmagent_remotewrite_errors_total climbs for the affected -remoteWrite.url.
• The same endpoint works from a browser or curl that trusts a corporate CA vmagent does not have.

Common Root Causes

• The server’s CA is not trusted — vmagent runs in a minimal image with no corporate/internal CA bundle installed.
• A self-signed certificate — the endpoint presents a cert with no external chain of trust.
• Hostname/SAN mismatch — the certificate is otherwise valid but its SAN does not match the host in -remoteWrite.url (this may also surface as a name-mismatch x509 error).
• An expired certificate — the chain was valid but the leaf or an intermediate has passed its notAfter date.

How to diagnose

Inspect the certificate the endpoint actually serves, including its issuer and SANs:

echo | openssl s_client -connect metrics.example.com:443 -servername metrics.example.com 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates -ext subjectAltName

Test whether the chain verifies against a specific CA file the way vmagent would:

curl -v --cacert /etc/vmagent/ca.crt 'https://metrics.example.com/api/v1/write'

Check what CA and server name vmagent is currently configured with:

ps aux | grep vmagent | grep -oE '\-remoteWrite.tls[^ ]*'

Fixes

1. Point vmagent at the CA with -remoteWrite.tlsCAFile. Supply the CA (or full chain) that issued the server certificate:

./vmagent -remoteWrite.url='https://metrics.example.com/api/v1/write' \
  -remoteWrite.tlsCAFile='/etc/vmagent/ca.crt'

For multiple remote_write targets, the tls* flags are positional — order them to match each -remoteWrite.url.

2. Set -remoteWrite.tlsServerName to match the certificate SAN. When you connect via an IP or a name that differs from the cert’s SAN, tell vmagent which name to verify:

./vmagent -remoteWrite.url='https://10.0.0.20/api/v1/write' \
  -remoteWrite.tlsCAFile='/etc/vmagent/ca.crt' \
  -remoteWrite.tlsServerName='metrics.example.com'

3. Renew an expired cert. If -dates shows the certificate has expired, reissue it on the server side; no client flag can make vmagent trust an expired chain.

4. As a last resort in trusted networks, use -remoteWrite.tlsInsecureSkipVerify=true. This disables verification entirely and should only ever be used on a private, controlled network while you fix the real trust problem:

./vmagent -remoteWrite.url='https://metrics.example.com/api/v1/write' \
  -remoteWrite.tlsInsecureSkipVerify=true

What to watch out for

• tlsInsecureSkipVerify=true defeats the purpose of TLS and exposes you to interception; treat it as temporary, never a fix committed to production.
• The tls* flags are per-remoteWrite and positional; mixing up their order silently applies the wrong CA to the wrong endpoint.
• Missing an intermediate certificate looks identical to an untrusted root — provide the full chain in the CA file if verification still fails.
• A cert that verifies today can expire later; monitor certificate expiry so remote_write does not break unattended.

Related

• VictoriaMetrics Error Guide: remote write connection refused
• VictoriaMetrics Error Guide: remote write 429
• VictoriaMetrics Error Guide: connection reset by peer