Tempo TraceQL Query Design Prompt
Write precise TraceQL queries to find slow, errored, or anomalous traces in Grafana Tempo — using span/resource attribute filters, structural operators, aggregates, and metrics-from-traces — instead of guessing in trace search.
- Target user
- Engineers debugging latency and errors in distributed traces with Tempo
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a distributed-tracing expert who writes TraceQL the way SREs write PromQL — surgically, to isolate exactly the traces that matter.
I will provide:
- The symptom (slow checkout, 5xx from a service, a specific tenant affected)
- The span/resource attributes my services emit (http.status_code, service.name, db.system, custom attrs)
- The Tempo version and whether TraceQL metrics are enabled
Your job:
1. **TraceQL mental model** — explain the `{ ... }` span-set selector, how a query returns traces whose spans match, and the difference between span-scoped (`span.`), resource-scoped (`resource.`), and intrinsic (`duration`, `status`, `name`, `kind`) fields. Note the difference vs. flat label matching.
2. **Write targeted queries** for my symptoms:
- Slow traces: `{ resource.service.name = "checkout" && duration > 2s }`
- Errored spans: filter on `status = error` and/or `span.http.status_code >= 500`
- Tenant/customer-scoped, plus combining conditions with `&&` / `||` and field existence checks.
3. **Structural operators** — use `>>` (descendant), `>` (child), `~` (sibling), and `&&` across spansets to express "a slow DB span UNDER a checkout request," which flat filters can't. Show concrete examples and the perf cost.
4. **Aggregates** — `count()`, `avg()`, `max()` over span attributes within a trace (e.g., traces with more than N retries, or total DB time > 500ms).
5. **TraceQL metrics** — if enabled, turn a trace query into a time series with `rate()` / `quantile_over_time()` (e.g., p99 latency of a specific operation) so it can drive a Grafana panel or alert. Note the version/feature-flag requirements.
6. **Performance & cost** — order filters most-selective-first, prefer intrinsics and indexed attributes, and warn which queries force full block scans.
7. **Saving for reuse** — turn the best queries into Grafana Explore links and dashboard panels.
Output as: (a) a ranked list of TraceQL queries per symptom with a one-line rationale, (b) the structural-operator examples, (c) any TraceQL-metrics expressions, (d) a perf note flagging expensive queries.
Bias toward: selective, fast queries; intrinsics over scanning; copy-pasteable expressions.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus Exemplars & Trace Correlation Prompt
Wire Prometheus exemplars end-to-end so a spike on a latency histogram links directly to the slow trace in Tempo — covering instrumentation, OpenMetrics exposition, storage, and Grafana exemplar links.
-
Grafana Tempo Distributed Tracing Prompt
Visualize traces in Grafana — Tempo data source, service graph, span metrics, trace search, OTLP integration.
-
Grafana k6 Load Test Metrics Dashboard Prompt
Stream k6 load-test results into Prometheus and build a Grafana dashboard that correlates virtual-user load, latency percentiles, error rates, and system saturation during a test run.
-
OpenTelemetry Tail Sampling Policy Design Prompt
Design an OpenTelemetry Collector tail-sampling policy that keeps every error and slow trace while cheaply down-sampling healthy traffic, and feeds clean span metrics into Prometheus.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.