🚨 Loki Error Guide: 'invalid tenant ID' — Sanitize the X-Scope-OrgID Value

Overview

Loki validates the tenant supplied in the X-Scope-OrgID header before it does anything with a request. When the value contains a reserved character or is too long, the distributor (on push) or querier (on read) rejects it with an HTTP 400 and an error like:

invalid tenant ID: '../evil' tenant IDs can only contain characters a-z, A-Z, 0-9, and !-_.*'()

You will also see variants such as:

tenant ID 'team|prod' contains unsupported character '|'
tenant ID is too long: max 150 characters

Because the tenant becomes a prefix in object storage paths (fake/, tenant-a/, and so on), Loki forbids path separators (/, \), the pipe |, and other reserved characters, and caps the length at roughly 150 characters. This keeps a malicious or malformed tenant from escaping its storage prefix or colliding with another tenant. This is distinct from no org id, which means the X-Scope-OrgID header is missing entirely — here the header is present but its value is not allowed.

Symptoms

• Push or query requests return HTTP 400 with invalid tenant ID naming the offending value.
• A tenant that contains /, \, or | fails while a plain alphanumeric tenant on the same cluster works.
• Promtail/Alloy logs server returned HTTP status 400 Bad Request (invalid tenant ID ...) and stops pushing for that client.
• Requests started failing right after an upstream system began deriving the tenant from a path, URL, or email address.
• A very long tenant (an ID concatenated with metadata) is rejected with tenant ID is too long.

Common Root Causes

• Path separators in the tenant — a value like team/prod or ../evil derived from a filesystem path or URL segment.
• Pipe or other reserved characters — team|prod, a,b, or similar, often produced by joining fields.
• Over-length tenant — an external identifier (UUID plus suffixes) exceeding the ~150 character cap.
• Unsanitized tenant from an upstream system — an ID copied straight from an SSO subject, email, or ticket key without normalization.
• Copy-paste whitespace or control characters — invisible characters that are not in the allowed set.

How to diagnose

1. Reproduce with a known-bad and known-good tenant so you can see exactly which value Loki rejects:

   # Fails: reserved character in the tenant
   curl -v -H "X-Scope-OrgID: team/prod" \
     "http://loki-gateway/loki/api/v1/labels"
   
   # Succeeds: sanitized tenant slug
   curl -v -H "X-Scope-OrgID: team-prod" \
     "http://loki-gateway/loki/api/v1/labels"

2. Inspect the raw header value the gateway forwards — check for hidden characters that the logs may not render:

   printf '%s' "team|prod" | openssl base64
   # decode on the other side to confirm what actually arrived

3. Find who sets the tenant — trace it back to the client or gateway route that injects X-Scope-OrgID:

   kubectl get configmap loki-gateway -o yaml \
     | grep -i 'X-Scope-OrgID'

4. Check the length — long external identifiers overflow the cap:

   printf '%s' "$TENANT" | wc -c   # must be <= ~150

5. Confirm the allowed-character contract in your Loki version’s limits before you pick a slug format:

   # Allowed set (Loki tenant IDs): a-z A-Z 0-9 and ! - _ . * ' ( )
   # Forbidden examples: / \ | , : whitespace

Fixes

Sanitize the X-Scope-OrgID value to the allowed character set at the point where it is first set. Replace anything outside a-zA-Z0-9 and !-_.*'() with a hyphen so team/prod becomes team-prod:

# normalize an arbitrary identifier into a safe tenant slug
raw="team/prod|us-east"
safe=$(printf '%s' "$raw" | tr -c 'a-zA-Z0-9._-' '-')
echo "$safe"   # team-prod-us-east

Map external identifiers to stable, short tenant slugs at the gateway rather than passing raw IDs through. Keep a lookup so the same external identity always yields the same safe tenant:

# nginx gateway: inject a curated tenant per authenticated route
map $ssl_client_s_dn $loki_tenant {
    default            "tenant-a";
    "~CN=team-prod"    "team-prod";
}
proxy_set_header X-Scope-OrgID $loki_tenant;

Keep the tenant short and within the length cap by hashing long external identifiers down to a fixed-width slug instead of concatenating metadata:

# derive a short, stable slug from a long external ID
long="6f1c...-prod-us-east-1-team-observability-platform-...."
short="t-$(printf '%s' "$long" | sha1sum | cut -c1-12)"
echo "$short"   # t-9a3b7c1e2f04  (well under 150 chars)

Set the sanitized tenant on the log client so writes carry a valid value and never rely on the raw upstream string:

loki.write "default" {
  endpoint {
    url       = "http://loki-gateway/loki/api/v1/push"
    tenant_id = "team-prod"
  }
}

Reject bad tenants early at the gateway with a clear message, so a malformed identifier never reaches Loki and callers get an actionable error instead of a raw 400:

# refuse tenants containing path separators or pipes before proxying
if ($http_x_scope_orgid ~ "[/\\|]") {
    return 400 "invalid tenant: reserved character";
}

What to watch out for

• invalid tenant ID is not no org id — one means the value is disallowed, the other means the header is missing. Adding a header will not help if the header you add is malformed.
• Changing a tenant slug changes its storage prefix, so an existing tenant’s historical data lives under the old name; migrating slugs orphans old data unless you plan for it.
• Sanitizing on only one caller is not enough — every client and gateway route that sets X-Scope-OrgID must produce the same safe value, or writes and reads land in different tenants.
• The length cap counts bytes, so multibyte characters consume the budget faster than they appear to; prefer plain ASCII slugs.
• Hashing external IDs makes tenants opaque; keep the mapping documented so operators can tell which real team a slug belongs to.

Related

• Loki Error Guide: ‘no org id’ — the sibling case where the X-Scope-OrgID header is missing rather than malformed.
• Loki Error Guide: ‘ingestion rate limit exceeded’ — per-tenant limits that apply once a valid tenant is accepted.
• Loki Error Guide: ‘maximum active stream limit exceeded’ — another per-tenant guardrail keyed on the tenant you supply.