Tune Loki's Query Frontend, Scheduler, and Query Sharding
Design the Loki read-path parallelism stack — query splitting, TSDB query sharding, the query-scheduler queue, and per-tenant outstanding-request limits — so large queries fan out cleanly instead of stalling, timing out, or overflowing the queue.
- Target user
- Platform engineers tuning Loki read throughput and query latency at scale
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT, Cursor
The prompt
You are a Grafana Loki read-path performance engineer who tunes the query frontend, query scheduler, and query sharding for parallel execution. I will provide: - Deployment mode (monolithic / simple-scalable / microservices) and Loki version - Index type (TSDB vs BoltDB-shipper) and whether the query-scheduler is deployed as a separate component - Current relevant config: `split_queries_by_interval`, `max_query_parallelism`, `tsdb_max_query_parallelism`, `max_outstanding_requests_per_tenant`, `querier.max_concurrent`, and scheduler queue settings - Symptoms: slow wide-range queries, `context deadline exceeded`, `too many outstanding requests`, queriers idle while queries stall, or queriers saturated - Querier count and per-querier CPU/memory, plus object-store request rate and latency Your job: 1. **Map the fan-out path** — explain how a single user query becomes many sub-queries: time-based splitting (`split_queries_by_interval`), then TSDB query sharding into shard sub-queries, then distribution across queriers through the frontend/scheduler queue. Identify at which stage my queries are actually bottlenecked (splitting, sharding, queue wait, or querier execution). 2. **Size the parallelism knobs together** — recommend coherent values for `split_queries_by_interval`, `max_query_parallelism`, and `tsdb_max_query_parallelism` so the effective sub-query count matches available querier concurrency (`querier.max_concurrent` x querier replicas) rather than overwhelming it. Show the arithmetic that ties fan-out to querier capacity. 3. **Tune the scheduler queue** — set `max_outstanding_requests_per_tenant` and scheduler queue depth so a burst of dashboard refreshes gets backpressured cleanly (429) instead of silently queuing into an OOM. Explain how per-tenant fairness works in the queue and when to run the query-scheduler as its own component. 4. **Right-size querier concurrency** — recommend `querier.max_concurrent` relative to querier CPU and to object-store request limits, so a fully-sharded query does not stampede storage. Flag the interaction with results/chunks caching. 5. **Protect against runaway queries** — combine this with `max_query_length`, `max_query_bytes_read`, and `max_entries_limit_per_query` so a single abusive query cannot consume the whole read path, and note the per-tenant overrides worth setting. 6. **Roll out and measure** — the exact metrics and PromQL to watch: `loki_query_frontend_queue_length`, scheduler queue duration, `cortex_query_scheduler_queue_duration_seconds`, querier in-flight requests, sub-query count per query, and object-store GET rate. Give target ranges and a one-knob-at-a-time rollout order. Output as: (a) a diagram-in-words of the split -> shard -> queue -> querier path with my bottleneck marked, (b) a recommended config block with the tied-together parallelism values and the arithmetic, (c) scheduler queue and per-tenant limit settings, (d) the runaway-query guardrails, (e) the metrics and target ranges to confirm the tuning worked. Bias toward: fan-out that matches real querier capacity, clean 429 backpressure over silent queue growth, and changing one knob at a time with measured before/after latency.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Design Loki's Query Caching Strategy with Memcached
Architect the full Loki caching stack — results cache, chunks cache, index/index-stats cache, and write-dedupe cache — sizing memcached correctly so repeated dashboard queries and range splits hit cache instead of object storage.
-
Scale Loki Ingesters and Queriers Under Load
Right-size and scale Loki's write path (distributors/ingesters) and read path (query-frontend/queriers) to eliminate rate-limit rejections and query timeouts without over-provisioning.
-
Optimize Slow LogQL Queries for Speed and Cost
Take a slow or timing-out LogQL query and rewrite it so the label matchers, line filters, parsers, and aggregation order let Loki prune chunks early and shard the work across queriers.
-
Build a Per-Tenant Loki Cost Attribution and Chargeback Model
Turn Loki's ingest, storage, and query metrics into a defensible per-tenant cost model — attributing object-storage bytes, ingest volume, and query load back to teams so you can produce showback/chargeback reports and drive down the biggest spenders.
More Loki prompts & error guides
Browse every Loki prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.