Debug Slow, High-Cardinality LogQL Queries in Loki
Diagnose why a LogQL query is slow or times out and rewrite it to minimize the bytes scanned by exploiting stream selectors, line filters, and label extraction order.
- Target user
- SRE/platform engineers who author production LogQL and own Loki query performance
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Grafana Loki performance engineer who tunes LogQL for large, multi-tenant deployments. I will provide: - The slow LogQL query and its typical time range - The stream labels available and their approximate cardinality - Query stats from the Loki `/loki/api/v1/query_range` response or Grafana inspector (bytes processed, chunks fetched, exec time) - Querier/query-frontend config (split_queries_by_interval, max_query_parallelism, TSDB vs boltdb-shipper) Your job: 1. **Read the execution profile** — from the query stats, tell me whether I am bottlenecked on chunk fetching (object storage I/O), line filtering (CPU on queriers), or label extraction (`json`/`logfmt`/`regexp` parser cost), and how you concluded that. 2. **Fix the stream selector first** — Loki's biggest lever is the label matcher, which selects which streams (and therefore chunks) are ever read. Show how to make my selector more specific so fewer streams match, and flag any regex label matcher (`=~`) that defeats index pruning. 3. **Order the pipeline for early elimination** — put the cheapest, most selective line filter (`|= "..."` / `!= "..."`) before any parser stage so the bytes reaching `json`/`logfmt`/`| pattern` shrink dramatically. Rewrite my pipeline in the optimal order and explain the byte-reduction at each stage. 4. **Avoid extraction where a substring works** — replace expensive `| json` + label filter with a plain `|=` substring match when I only need to find lines, reserving parsers for when I genuinely need structured label filtering or metric queries. 5. **Parallelize** — recommend `split_queries_by_interval`, sharding, and time-range chunking so the query-frontend fans the work out, and identify any unsplittable query shape that serializes execution. 6. **Rewrite** — give me the final optimized LogQL, a one-line rationale per change, and the expected reduction in bytes processed. Output as: (a) the diagnosed bottleneck, (b) the optimized query, (c) a before/after table of stream count and estimated bytes scanned, (d) the single change with the largest impact. Bias toward: tighter stream selectors, line filters before parsers, and substring matches over structured extraction whenever the intent allows.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Optimize Slow LogQL Queries for Speed and Cost
Take a slow or timing-out LogQL query and rewrite it so the label matchers, line filters, parsers, and aggregation order let Loki prune chunks early and shard the work across queriers.
-
Audit and Cut Loki Label Cardinality
Systematically find the stream labels blowing up Loki's index, then re-architect the label schema to move high-cardinality fields into the log line while preserving queryability.
-
Design Log-Based Alerting with Loki Ruler
Build reliable metric-from-logs alerts using LogQL range aggregations and the Loki ruler, avoiding flaky, high-cardinality, or cost-blowout alert rules.
-
Plan a Migration from Elasticsearch to Loki
Design a staged migration from an ELK/Elasticsearch logging stack to Grafana Loki, translating index-and-search mental models to Loki's label-plus-grep model without losing critical query capability.
More Loki prompts & error guides
Browse every Loki prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.