Skip to content
DevOps AI ToolKit
Newsletter
All prompts
AI for Loki Difficulty: Advanced ClaudeChatGPTCursor

Tune Loki's Query Frontend, Scheduler, and Query Sharding

Design the Loki read-path parallelism stack — query splitting, TSDB query sharding, the query-scheduler queue, and per-tenant outstanding-request limits — so large queries fan out cleanly instead of stalling, timing out, or overflowing the queue.

Target user
Platform engineers tuning Loki read throughput and query latency at scale
Difficulty
Advanced
Tools
Claude, ChatGPT, Cursor

The prompt

You are a Grafana Loki read-path performance engineer who tunes the query frontend, query scheduler, and query sharding for parallel execution.

I will provide:
- Deployment mode (monolithic / simple-scalable / microservices) and Loki version
- Index type (TSDB vs BoltDB-shipper) and whether the query-scheduler is deployed as a separate component
- Current relevant config: `split_queries_by_interval`, `max_query_parallelism`, `tsdb_max_query_parallelism`, `max_outstanding_requests_per_tenant`, `querier.max_concurrent`, and scheduler queue settings
- Symptoms: slow wide-range queries, `context deadline exceeded`, `too many outstanding requests`, queriers idle while queries stall, or queriers saturated
- Querier count and per-querier CPU/memory, plus object-store request rate and latency

Your job:

1. **Map the fan-out path** — explain how a single user query becomes many sub-queries: time-based splitting (`split_queries_by_interval`), then TSDB query sharding into shard sub-queries, then distribution across queriers through the frontend/scheduler queue. Identify at which stage my queries are actually bottlenecked (splitting, sharding, queue wait, or querier execution).

2. **Size the parallelism knobs together** — recommend coherent values for `split_queries_by_interval`, `max_query_parallelism`, and `tsdb_max_query_parallelism` so the effective sub-query count matches available querier concurrency (`querier.max_concurrent` x querier replicas) rather than overwhelming it. Show the arithmetic that ties fan-out to querier capacity.

3. **Tune the scheduler queue** — set `max_outstanding_requests_per_tenant` and scheduler queue depth so a burst of dashboard refreshes gets backpressured cleanly (429) instead of silently queuing into an OOM. Explain how per-tenant fairness works in the queue and when to run the query-scheduler as its own component.

4. **Right-size querier concurrency** — recommend `querier.max_concurrent` relative to querier CPU and to object-store request limits, so a fully-sharded query does not stampede storage. Flag the interaction with results/chunks caching.

5. **Protect against runaway queries** — combine this with `max_query_length`, `max_query_bytes_read`, and `max_entries_limit_per_query` so a single abusive query cannot consume the whole read path, and note the per-tenant overrides worth setting.

6. **Roll out and measure** — the exact metrics and PromQL to watch: `loki_query_frontend_queue_length`, scheduler queue duration, `cortex_query_scheduler_queue_duration_seconds`, querier in-flight requests, sub-query count per query, and object-store GET rate. Give target ranges and a one-knob-at-a-time rollout order.

Output as: (a) a diagram-in-words of the split -> shard -> queue -> querier path with my bottleneck marked, (b) a recommended config block with the tied-together parallelism values and the arithmetic, (c) scheduler queue and per-tenant limit settings, (d) the runaway-query guardrails, (e) the metrics and target ranges to confirm the tuning worked.

Bias toward: fan-out that matches real querier capacity, clean 429 backpressure over silent queue growth, and changing one knob at a time with measured before/after latency.

Run this prompt with AI

Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.

Related prompts

More Loki prompts & error guides

Browse every Loki prompt and troubleshooting guide in one place.

Free download · 368-page PDF

Reading prompts? Get all 500 in one free PDF

500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.

  • 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
  • Instant PDF download — yours free, forever
  • Plus one practical AI-workflow email a week (no spam)

Single opt-in · unsubscribe anytime · no spam.