🚨 Grafana Error Guide: 'Invalid API key' / 401 Unauthorized — Fix Grafana HTTP API Auth

Overview

Grafana’s HTTP API authenticates callers with a bearer token. When the token is missing, malformed, expired, deleted, or belongs to the wrong organization, Grafana rejects the request with 401 Unauthorized and a JSON body. When the token is valid but lacks the required role, you get 403 instead — an important distinction that tells you whether the problem is authentication or authorization.

The literal responses you’ll see:

{"message":"Invalid API key"}
{"message":"Unauthorized"}
{"message":"Expired API key"}
{"message":"Permission denied"}

Note that legacy API keys are deprecated in favor of Service Accounts and their tokens (prefixed glsa_) since Grafana 9. Both are sent the same way — as a Authorization: Bearer <token> header — but new automation should use service account tokens, which have their own role, org scope, and optional expiry.

Symptoms

• API calls return HTTP 401 with {"message":"Invalid API key"} or {"message":"Unauthorized"}.
• A token that worked yesterday now returns {"message":"Expired API key"}.
• Reads succeed but writes return 403 {"message":"Permission denied"}.
• The same token works against localhost:3000 but fails through the load balancer / ingress.
• Terraform, CI jobs, or the Grafana provisioning API suddenly fail to authenticate.

Common Root Causes

1. Missing or malformed Authorization header

The header must be exactly Authorization: Bearer <token>. A missing Bearer prefix, a stray newline, or using -H "Authorization: <token>" all yield 401.

# WRONG - no Bearer prefix
curl -H "Authorization: glsa_xxxxxxxx" http://localhost:3000/api/dashboards/home
# {"message":"Unauthorized"}

# RIGHT
curl -H "Authorization: Bearer glsa_xxxxxxxx" http://localhost:3000/api/dashboards/home

{"message":"Unauthorized"}

A common shell trap: a token read from a file with a trailing newline. Use $(cat token | tr -d '\n').

2. Expired or deleted token

Service account tokens can be created with a TTL. Once past it, Grafana returns a distinct message.

{"message":"Expired API key"}

If the key was revoked or the whole service account deleted, you get Invalid API key instead. List active service accounts and tokens to confirm the key still exists.

3. Wrong organization

API keys and service accounts are scoped to a single org. A token minted in org 1 returns 401 against a resource in org 2. Multi-org setups must either mint per-org tokens or set the org context.

{"message":"Invalid API key"}

4. Using Basic auth where a token is expected (or vice versa)

Some endpoints and setups expect Authorization: Bearer glsa_...; passing -u admin:password (Basic auth) may be disabled or mismatched.

# Basic auth - works only if enabled and creds are valid
curl -u admin:admin http://localhost:3000/api/org
# Token auth
curl -H "Authorization: Bearer glsa_xxxx" http://localhost:3000/api/org

5. Reverse proxy stripping the Authorization header

Nginx, Traefik, or an ELB may drop or overwrite the Authorization header, so Grafana never sees the token and returns 401. This is the classic “works on localhost, fails through the proxy” symptom.

location / {
  proxy_pass http://grafana:3000;
  proxy_set_header Authorization $http_authorization;  # must be preserved
  proxy_pass_request_headers on;
}

6. Valid token, insufficient role → 403 not 401

A Viewer token can read but not create. This returns 403, not 401 — meaning auth succeeded but the role is too low.

{"message":"Permission denied"}

Assign the service account an Editor or Admin role for write operations.

Diagnostic Workflow

Step 1 — Capture the exact status code and body. 401 vs 403 decides everything.

curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer glsa_xxxx" \
  http://localhost:3000/api/dashboards/home
# 401 -> auth problem   403 -> role problem   200 -> fine

Step 2 — Test from the Grafana host directly, bypassing the proxy, to isolate header stripping.

curl -s -H "Authorization: Bearer glsa_xxxx" http://localhost:3000/api/user | jq .

Step 3 — Verify the token/service account exists and its role.

curl -s -u admin:admin http://localhost:3000/api/serviceaccounts/search | jq '.serviceAccounts[] | {id,name,role,isDisabled}'

Step 4 — Mint a fresh service account token to rule out expiry/revocation.

# Create service account
curl -s -u admin:admin -H "Content-Type: application/json" \
  -d '{"name":"ci-bot","role":"Editor"}' \
  http://localhost:3000/api/serviceaccounts
# Create a token on it (returns glsa_...)
curl -s -u admin:admin -H "Content-Type: application/json" \
  -d '{"name":"ci-token","secondsToLive":0}' \
  http://localhost:3000/api/serviceaccounts/2/tokens | jq .key

Step 5 — Confirm the proxy forwards the header by comparing localhost vs the public URL.

curl -s -o /dev/null -w "localhost:%{http_code}\n" -H "Authorization: Bearer glsa_xxxx" http://localhost:3000/api/user
curl -s -o /dev/null -w "proxy:%{http_code}\n"     -H "Authorization: Bearer glsa_xxxx" https://grafana.example.com/api/user

Example Root Cause Analysis

A CI pipeline that provisioned dashboards started failing with 401 {"message":"Unauthorized"}. The token worked when an engineer curled Grafana directly on the node but failed from the runner, which went through an Nginx ingress. Testing with curl -w "%{http_code}" confirmed 200 on localhost:3000 and 401 through https://grafana.example.com. The ingress had been re-templated and lost its proxy_set_header Authorization $http_authorization; line, so Grafana never received the bearer token. Restoring the header forwarding fixed every call immediately. Root cause: a reverse proxy stripping the Authorization header — an authentication failure that had nothing to do with the token itself.

Prevention Best Practices

• Migrate off legacy API keys to service account tokens (glsa_) — they support roles, org scope, and rotation.
• Set sensible token expiry and rotate before it lapses; alert on Expired API key in logs.
• Store tokens in a secret manager, inject at runtime, and strip trailing newlines.
• Grant least privilege: Viewer for reads, Editor for dashboards, Admin only when required.
• Test the full path (through the proxy) in CI, not just localhost, so header stripping is caught early.
• Explicitly preserve Authorization in every proxy/ingress config in front of Grafana.

Quick Command Reference

# Status code only - distinguishes 401 vs 403 vs 200
curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer glsa_xxxx" \
  http://localhost:3000/api/dashboards/home

# Who am I (validates the token)
curl -s -H "Authorization: Bearer glsa_xxxx" http://localhost:3000/api/user | jq .

# List service accounts and roles
curl -s -u admin:admin http://localhost:3000/api/serviceaccounts/search | jq .

# Create a service account + token
curl -s -u admin:admin -H "Content-Type: application/json" \
  -d '{"name":"ci-bot","role":"Editor"}' http://localhost:3000/api/serviceaccounts
curl -s -u admin:admin -H "Content-Type: application/json" \
  -d '{"name":"ci-token","secondsToLive":0}' \
  http://localhost:3000/api/serviceaccounts/2/tokens | jq .key

# Compare localhost vs proxy to catch stripped headers
curl -s -o /dev/null -w "local:%{http_code}\n" -H "Authorization: Bearer glsa_xxxx" http://localhost:3000/api/user
curl -s -o /dev/null -w "proxy:%{http_code}\n" -H "Authorization: Bearer glsa_xxxx" https://grafana.example.com/api/user

Conclusion

When the Grafana API returns Invalid API key or 401 Unauthorized, work through these causes:

1. Malformed Authorization header — missing Bearer prefix or trailing newline; format is exactly Authorization: Bearer <token>.
2. Expired or deleted token — Expired API key; mint a fresh service account token.
3. Wrong organization — token scoped to a different org returns 401; use a per-org token.
4. Wrong auth scheme — Basic vs Bearer mismatch; send the type the endpoint expects.
5. Reverse proxy stripping Authorization — works on localhost, fails through the proxy; preserve the header.
6. Insufficient role → 403, not 401 — the token authenticates but lacks permission; raise the service account role.

For more, see the Grafana category and the related backend guide on database connection failures.