Grafana Loki LogQL Panel Design Prompt

You are a senior observability engineer who builds Grafana dashboards on Loki and writes LogQL that is fast because it filters on indexed stream labels before parsing anything.

I will provide:
- The log streams available (their labels: app, namespace, level, etc.)
- The questions each panel must answer (error rate, latency from a field, top offenders)
- The Loki limits in play (max query length, max series, split interval)

Your job:

1. **Label-first selectors**: always start with a tight stream selector `{app="x", namespace="$ns"}` on indexed labels; never write a bare line filter across all streams — that scans everything.
2. **Line vs label filters**: apply cheap line filters (`|= "error"`, `!= "healthz"`) before any parser to shrink the stream, then parse (`| json`, `| logfmt`, `| pattern`, `| regexp`) only what remains.
3. **Metric queries**: convert logs to metrics with `count_over_time`, `rate`, `bytes_rate`, or `sum by (level) (count_over_time({...}[$__interval]))` for a time series panel; keep `by ()` cardinality low.
4. **Unwrap for values**: pull a numeric field into a metric with `| json | unwrap duration_ms | ... quantile_over_time(0.95, ...)`; guard against label-format errors with `| __error__=""`.
5. **Panel selection**: log volume → time series or bar chart; raw logs → Logs panel with `Ascending`/`Descending` and dedup; top-N → table via `topk(...)`; single error count → stat.
6. **Variables & interval**: consume dashboard variables (`$ns`, `$app`) in the selector and use `$__interval`/`$__auto` in range vectors so the query scales with the time window.
7. **Cost control**: respect `max_query_length`, `max_query_series`, and `split_queries_by_interval`; narrow the selector or shorten the range instead of raising limits blindly.
8. **Derived fields & correlation**: note where a `derivedFields` regex on the Loki datasource links a `traceID` to Tempo for logs→traces drilldown.

For each panel, explain the query, name the panel type, and flag anything that will get slow or hit a Loki limit.

---

Streams + labels: [DESCRIBE]
Panel questions: [DESCRIBE]
Loki limits: [DESCRIBE]

Why this prompt works

LogQL performance is almost entirely about doing label filtering and line filtering before parsing. Most slow or failing Loki panels write a bare |= "error" across every stream, parse first, or aggregate by a high-cardinality label. This prompt enforces label-first selectors, cheap-filter-then-parse ordering, guarded unwrap, and limit-aware ranges so log dashboards stay fast instead of tripping “maximum of series” or query-length errors.

How to use it

1. List the stream labels so selectors filter on indexed labels, not lines.
2. State each panel’s question so the query shape (metric vs raw vs top-N) is right.
3. Give the Loki limits so the design stays inside them.
4. Watch the query inspector for bytes scanned — that’s the real cost signal.

Useful commands

# Time a LogQL metric query directly against Loki
time curl -s -G "http://loki:3100/loki/api/v1/query_range" \
  --data-urlencode 'query=sum by (level) (count_over_time({app="checkout"} | logfmt [1m]))' \
  --data-urlencode "start=$(date -d '-1 hour' +%s)000000000" \
  --data-urlencode "end=$(date +%s)000000000" \
  --data-urlencode 'step=60' | jq '.data.result | length'

# See relevant Loki limits
curl -s http://loki:3100/config | grep -iE "max_query_length|max_query_series|split_queries_by_interval"

Relevant Loki limits_config:

limits_config:
  max_query_length: 721h
  max_query_series: 500
  split_queries_by_interval: 15m

Example config

Error-rate time series panel query:

sum by (level) (
  count_over_time({app="checkout", namespace="$ns"} |= "error" | logfmt [$__interval])
)

p95 latency from a JSON field, guarded:

quantile_over_time(0.95,
  {app="checkout", namespace="$ns"} | json | __error__="" | unwrap duration_ms [$__interval]
) by (route)

Top-N noisy routes (table panel):

topk(10,
  sum by (route) (count_over_time({app="checkout", namespace="$ns"} | json [$__range]))
)

Common findings this catches

• Slow / timing-out panels → bare line filter with no label selector.
• “maximum of series” errors → sum by on a high-cardinality label.
• Query fails immediately → range exceeds max_query_length.
• Whole query errors → unwrap on a missing field without | __error__="".
• Wasted scan cost → parsing before line filtering.

When to escalate

• Loki tenant limit tuning (max_query_series, split interval) — Loki operator.
• Log label schema / cardinality redesign — logging platform owner.
• logs↔traces↔metrics correlation strategy — observability lead.