🚨 VictoriaMetrics Error: 'cannot authorize request: missing Authorization request header' — Cause, Fix, and Troubleshooting Guide

Overview

vmauth is the VictoriaMetrics reverse proxy and auth gateway. It inspects every incoming request, matches it against a user in its -auth.config, and routes it to the right backend. When a client reaches vmauth (default port 8427) without valid credentials, vmauth rejects it with HTTP 401 and:

cannot authorize request: missing "Authorization" request header; the client must provide valid credentials

This is an authentication failure at the gateway, not a backend error — the request never reached vminsert or vmselect. Either the client sent no (or a malformed) Authorization header, or the credentials it sent do not correspond to any user defined in -auth.config.

Symptoms

• Clients receive HTTP 401 from the vmauth endpoint on port 8427.
• Prometheus/vmagent remote_write targets show authorization failures pointing at vmauth.
• Grafana datasources through vmauth return “cannot authorize request”.
• A previously working user starts getting 401 right after an -auth.config edit.
• curl without credentials fails, while the same URL against the backend directly (bypassing vmauth) succeeds.

Common Root Causes

• Client sends no or malformed Authorization header — the request omits Authorization, or the header value is truncated/garbled.
• User or bearer token not present in -auth.config — the credential is valid-looking but does not match any configured user.
• Wrong basic-auth or bearer value — a typo, stale password, or rotated token that no longer matches config.
• -auth.config edited but not reloaded — the new user exists on disk but vmauth is still running the old config in memory.

How to diagnose

Reproduce the request and watch the status code. A bare request should return 401:

# No credentials -> expect 401
curl -si 'http://localhost:8427/api/v1/query?query=up' | head -1

Now send the intended credentials and compare:

# Basic auth
curl -si -u 'tenant1:REDACTED' 'http://localhost:8427/api/v1/query?query=up' | head -1
# Bearer token
curl -si -H 'Authorization: Bearer REDACTED' 'http://localhost:8427/api/v1/query?query=up' | head -1

Confirm the user actually exists in the loaded config, and check vmauth’s own metrics to see whether the request is even being matched to a user:

grep -n 'username\|bearer_token\|url_prefix' /etc/vmauth/auth.yml
curl -s 'http://localhost:8427/metrics' | grep vmauth_user_requests_total

Fixes

1. Add the correct Authorization header from the client. Use Basic or Bearer depending on how the user is defined:

curl -u 'tenant1:REDACTED' 'http://localhost:8427/api/v1/query?query=up'
# or
curl -H 'Authorization: Bearer REDACTED' 'http://localhost:8427/api/v1/query?query=up'

2. Define the user in -auth.config with a url_prefix so vmauth knows where to route authorized requests:

users:
  - username: "tenant1"
    password: "REDACTED"
    url_prefix: "http://vmselect:8481/select/0/prometheus/"

3. Reload vmauth after editing the config. vmauth does not pick up changes automatically:

# Signal-based reload
kill -HUP "$(pidof vmauth)"
# Or via the reload endpoint
curl -s 'http://localhost:8427/-/reload'

4. Verify routing and matches through vmauth’s metrics so you can confirm the user is being recognized:

curl -s 'http://localhost:8427/metrics' | grep vmauth_user_requests_total

What to watch out for

• After any -auth.config change, reload vmauth — a missing reload is the most common cause of “it looks right but still 401”.
• Basic-auth vs bearer must match how the user is defined; sending a bearer token to a basic-auth user (or vice versa) still fails.
• Proxies and load balancers in front of vmauth can strip the Authorization header — verify it survives the whole path to port 8427.
• Keep credentials out of shell history and logs; rotate tokens through the config and reload rather than embedding them in URLs.

Related

• VictoriaMetrics Error Guide: missing tenant id
• VictoriaMetrics Error Guide: remote write 429
• VictoriaMetrics Error Guide: remote write connection refused