🚨 Loki Error Guide: 'has 25 label names; limit 15' — Cut Label Cardinality Before the Distributor Rejects Your Push

Overview

Loki’s distributor validates every stream label set as it ingests a push. When a stream carries more labels than the configured ceiling, the distributor rejects the push with an HTTP 400 and a message like this:

stream '{app="api", ...}' has 25 label names; limit 15

The limit comes from limits_config.max_label_names_per_series, which defaults to 15. Every unique combination of label names and values is a distinct stream, so a high label count is not just a validation failure — it is a warning that you are about to explode Loki’s stream cardinality. When the distributor drops these pushes, it increments loki_discarded_samples_total{reason="max_label_names_per_series"}, which is the metric to watch and alert on.

The fix is almost never to raise the limit. It is to stop your log agent from attaching high-cardinality labels like pod, container, node, ip, and request_id, which belong in the log line or in structured metadata, not in the stream label set.

Symptoms

• Pushes fail with HTTP 400 and has N label names; limit 15 in the distributor logs or in the agent’s send errors.
• loki_discarded_samples_total{reason="max_label_names_per_series"} climbs steadily.
• A subset of applications (usually the noisiest ones) lose logs while others ingest fine.
• Promtail/Alloy client metrics show increasing dropped or failed batches for specific targets.
• Stream and active-series counts trend upward right before the rejections begin.

Common Root Causes

• Log agent attaching per-request labels — the pipeline promotes request_id, trace_id, or session_id to labels, so every request becomes its own stream and the label count balloons.
• Kubernetes metadata over-labeling — pod, container, node, namespace, ip, and image are all promoted to labels when only namespace/app are stable enough to belong there.
• Copying every JSON field into labels — a stage.json or stage.labels block that maps the entire parsed structure into labels instead of a curated subset.
• Dynamic infrastructure labels — autoscaled node names, ephemeral IPs, or container IDs that change constantly and inflate both the label count and the stream count.
• Well-intentioned “add context” changes — someone adds a couple of labels for one debugging session and never removes them, pushing a busy stream over the ceiling.
• Multi-tenant defaults left unbounded — a shared limits_config where one tenant’s agent config is far more aggressive than the platform default assumes.

How to diagnose

1. Confirm the rejection reason in the distributor logs so you are not chasing a different validation error:

   kubectl logs -l app=loki,component=distributor -n loki --tail=300 \
     | grep 'label names; limit'

2. Quantify the discards by reason to confirm this is the dominant drop cause:

   sum by (reason) (
     rate(loki_discarded_samples_total{reason="max_label_names_per_series"}[5m])
   )

3. List the offending stream’s labels to see exactly which labels are being attached and which are high-cardinality:

   logcli series '{app="api"}' --analyze-labels \
     --addr=https://loki-gateway/ --org-id=tenant-a

4. Read the effective limit so you know the real ceiling in force for the tenant:

   limits_config:
     max_label_names_per_series: 15

5. Inspect the agent pipeline that produces the stream. In Promtail/Alloy, find the stages that set labels and identify which are stable (keep) versus dynamic (drop):

   pipeline_stages:
     - json:
         expressions:
           level: level
           request_id: request_id
     - labels:
         level:
         request_id:   # high-cardinality: this is the problem

Fixes

Drop high-cardinality labels in Promtail before sending — keep only stable, low-cardinality labels like namespace, app, and level, and remove per-request identifiers with labeldrop:

pipeline_stages:
  - labeldrop:
      - request_id
      - pod
      - container
      - ip
      - node

Do the same in Grafana Alloy with a relabel rule that keeps only the labels you actually query on:

loki.relabel "keep_stable" {
  forward_to = [loki.write.default.receiver]
  rule {
    action        = "labeldrop"
    regex         = "request_id|pod|container|ip|node|image_id"
  }
}

Move dynamic fields into the log line, not labels — leave request_id and friends in the JSON body and extract them at query time, so they cost nothing at ingest:

{namespace="prod", app="api"} | json | request_id="abc-123"

Use structured metadata for high-cardinality context — attach fields like trace_id as structured metadata rather than stream labels, which keeps cardinality flat while remaining queryable:

limits_config:
  allow_structured_metadata: true

Raise the limit only as a deliberate, temporary stopgap — bump it just enough to unblock ingestion while you fix the agent, and treat the raise as debt to pay down:

limits_config:
  max_label_names_per_series: 20   # temporary; reduce after agent cleanup

What to watch out for

• Raising max_label_names_per_series does not fix cardinality — it lets more streams through, which can quietly overload ingesters and blow up your index and memory.
• request_id, trace_id, ip, and timestamps are classic anti-pattern labels; they should live in the log body or structured metadata, never in the stream label set.
• A per-tenant override can mask the problem for one tenant while the platform default still bites everyone else; keep tenant limits explicit and reviewed.
• Dropped pushes are unrecoverable — the agent may retry, but if the label set is still over the limit those retries also fail and the logs are lost.
• Watch loki_discarded_samples_total{reason="max_label_names_per_series"} continuously; a sudden rise usually means someone shipped a new labeling change.

Related

• Loki Error Guide: ‘Maximum active stream limit exceeded’ — the stream-count blowup that too many labels directly causes.
• Loki Error Guide: ‘Per stream rate limit exceeded’ — the throughput side of the same over-labeling problem.
• Loki Error Guide: ‘maximum of series reached’ — the query-time cost of exploded cardinality.