Audit and Cut Loki Label Cardinality
Systematically find the stream labels blowing up Loki's index, then re-architect the label schema to move high-cardinality fields into the log line while preserving queryability.
- Target user
- Platform engineers responsible for Loki index health and ingester stability
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Grafana Loki architect who specializes in taming stream/label cardinality. I will provide: - Output of `logcli series` or `/loki/api/v1/series` for a representative time window - My current label set and which pipeline stages (promtail/Alloy/OTel) attach them - Ingester metrics: `loki_ingester_memory_streams`, active streams per tenant, any OOMs or `per-user streams limit exceeded` errors - The queries teams actually run against these logs Your job: 1. **Rank the offenders** — estimate the cardinality contribution of each label (approximate distinct value count) and identify the ones multiplying stream count: typically `pod`, `pod_ip`, `request_id`, `trace_id`, `container_id`, or timestamps accidentally promoted to labels. 2. **Apply the core rule** — labels are for low-cardinality dimensions you filter/aggregate streams by (namespace, app, env, level, cluster); everything high-cardinality belongs in the log line, extracted at query time with `| json` / `| logfmt` / `| pattern`. For each offending label, decide keep, drop, or demote-to-line, with the reason. 3. **Show the relabel config** — write the promtail/Grafana Alloy relabeling (or OTel processor) that drops the label from the stream and ensures the value is still present in the log body for query-time extraction. 4. **Preserve queries** — for every demoted label, give the LogQL rewrite that reproduces the old query using a line filter or parser + label filter, so teams lose no capability. 5. **Quantify** — estimate the new active-stream count and the reduction in index size and ingester memory, and set `max_streams_per_user` / `max_label_names_per_series` limits that would have caught this earlier. 6. **Guardrail** — recommend an alert on stream growth rate and a CI check that rejects new high-cardinality labels in the pipeline config. Output as: (a) ranked cardinality table, (b) keep/drop/demote decision per label, (c) the relabel/processor config, (d) LogQL rewrites for demoted labels, (e) the projected stream-count and memory reduction. Bias toward: aggressively low-cardinality labels, demoting to the line over dropping data entirely, and enforceable limits that prevent regression.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Adopt Structured Metadata to Cut Loki Label Cardinality
Move high-cardinality fields like trace_id, request_id, and pod out of stream labels into Loki 3.x structured metadata, so they stay queryable without multiplying streams or index size.
-
Build a Per-Tenant Loki Cost Attribution and Chargeback Model
Turn Loki's ingest, storage, and query metrics into a defensible per-tenant cost model — attributing object-storage bytes, ingest volume, and query load back to teams so you can produce showback/chargeback reports and drive down the biggest spenders.
-
Tune Loki Chunk Sizing and Object Storage Layout
Optimize Loki chunk targets, compression, and object-storage configuration to balance query speed, storage cost, and request rate against S3/GCS/Azure limits.
-
Debug Slow, High-Cardinality LogQL Queries in Loki
Diagnose why a LogQL query is slow or times out and rewrite it to minimize the bytes scanned by exploiting stream selectors, line filters, and label extraction order.
More Loki prompts & error guides
Browse every Loki prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.