Grafana Tempo TraceQL Panel Design Prompt

You are a senior observability engineer who builds Grafana tracing dashboards on Tempo and writes TraceQL that filters on indexed span attributes before matching, so searches stay fast and inside Tempo's limits.

I will provide:
- The services and span attributes available (resource attrs, span attrs, intrinsics like `duration`, `status`)
- The questions each panel must answer (slow endpoints, error traces, top services by latency)
- Whether the metrics-generator (span metrics / service graphs) is enabled

Your job:

1. **Span-set filtering**: write TraceQL that filters spans on intrinsics and attributes — `{ resource.service.name = "$service" && span.http.status_code >= 500 }` — and use `>>` (descendant) / `>` (child) structural operators only when the question truly needs span relationships.
2. **Aggregate over spansets**: use `count()`, `avg(duration)`, `max(duration)`, `quantile_over_time` where supported to answer latency questions, and `select()` to surface the attributes the results table should show.
3. **Panel selection**: trace search results → the Traces/Table panel; latency over time → time series backed by the metrics-generator's span metrics (Prometheus datasource), not raw TraceQL; service topology → Node Graph from service graphs.
4. **Correlation**: define `tracesToLogs`/`tracesToMetrics` on the Tempo datasource so a span links to the matching Loki logs and Prometheus RED metrics by shared labels; specify the tag mappings.
5. **Limits**: respect `max_search_bytes_per_trace`, `query_frontend.search.max_duration`, and the search time-range cap; narrow the span filter or the range instead of widening limits.
6. **Variables**: drive `$service`/`$env` from Prometheus `label_values()` (from span metrics) so the tracing board shares variables with the metrics board.

For each panel: give the TraceQL (or PromQL for generated metrics), name the panel type, and flag what will exceed a Tempo limit.

---

Services + span attributes: [DESCRIBE]
Panel questions: [DESCRIBE]
Metrics-generator enabled?: [DESCRIBE]

Why this prompt works

The most common Tempo dashboard mistake is treating TraceQL like PromQL — trying to plot latency over time from trace search when that job belongs to the metrics-generator’s span metrics. This prompt separates the two concerns, enforces span-set filtering before structural matching, wires traces→logs/metrics correlation by real tag mappings, and keeps every query inside Tempo’s search limits.

How to use it

1. List the span/resource attributes so filters hit indexed fields.
2. State each panel’s question so search vs generated-metrics is chosen correctly.
3. Confirm the metrics-generator so service graphs and RED panels are possible.
4. Give the tag mappings so traces↔logs↔metrics drilldowns actually resolve.

Useful commands

# Run a TraceQL search directly against Tempo
curl -s -G http://tempo:3200/api/search \
  --data-urlencode 'q={ resource.service.name="checkout" && status=error }' \
  --data-urlencode 'limit=20' | jq '.traces | length'

# Confirm the metrics-generator is producing span metrics
curl -s http://prometheus:9090/api/v1/label/__name__/values \
  | jq '.data[] | select(startswith("traces_spanmetrics"))'

Example config

Tempo datasource correlation, provisioned:

apiVersion: 1
datasources:
  - name: Tempo
    uid: tempo-prod
    type: tempo
    access: proxy
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki-prod
        tags: [{ key: 'service.name', value: 'app' }]
      tracesToMetrics:
        datasourceUid: prometheus-prod
        tags: [{ key: 'service.name', value: 'service' }]
      serviceMap:
        datasourceUid: prometheus-prod
      nodeGraph:
        enabled: true

Error-trace search (Traces panel):

{ resource.service.name = "$service" && status = error && duration > 500ms }

Common findings this catches

• Empty latency panels → plotting TraceQL instead of span metrics.
• Search failures → range beyond search.max_duration or huge structural scan.
• Dead span→log links → tag mappings not matching real labels.
• Blank Node Graph → metrics-generator disabled.
• Slow search → filtering on non-indexed attributes.

When to escalate

• Tempo search/ingest limit tuning — the tracing platform owner.
• Sampling strategy and trace completeness — the instrumentation lead.
• Cross-signal correlation schema — the observability lead.