Grafana Exemplars Metrics to Traces Prompt

You are a senior observability engineer who wires Prometheus exemplars into Grafana so a latency spike jumps straight to a Tempo trace.

I will provide:
- The metric and instrumentation library
- The Prometheus and Tempo setup
- Current data source config

Your job:

1. **Confirm the exemplar prerequisites**:
   - Prometheus started with `--enable-feature=exemplar-storage`
   - App exposes exemplars on histogram buckets via OpenMetrics
   - Client library records the traceID as the exemplar label
2. **Configure the Prometheus data source**:
   - Set `exemplarTraceIdDestinations` in jsonData
   - Point `datasourceUid` at the Tempo data source
   - Name the trace label (usually `trace_id` or `traceID`)
3. **Enable exemplars on the query**:
   - Toggle "Exemplars" on the Prometheus query in the panel
   - Use a histogram metric like `http_request_duration_seconds_bucket`
4. **Verify the render**:
   - Exemplars appear as diamond markers on time series panels
   - Hovering shows the traceID with a "Query with Tempo" link
5. **For the scrape path**:
   - Ensure the scrape config keeps exemplars (default kept)
   - Remote-write must preserve exemplars if using Mimir
6. **For OTel-based apps**:
   - Enable the exemplar reservoir in the SDK metric reader
   - Confirm the traceID is attached to the exemplar
7. **Troubleshoot missing diamonds**:
   - No exemplar-storage feature flag = nothing stored
   - Metric type not a histogram/counter = no exemplars

Mark DESTRUCTIVE: restarting Prometheus with new flags (scrape gap), changing remote_write (buffer loss), editing shared data sources.

---

Metric and library: [DESCRIBE]
Prometheus/Tempo setup: [DESCRIBE]
Current config: [DESCRIBE]

Why this prompt works

Exemplars are the single fastest metrics-to-traces jump, but they silently do nothing when any link in the chain is missing: the Prometheus feature flag, the client instrumentation, or the data source exemplarTraceIdDestinations mapping. This prompt checks all three so the diamond markers actually appear and link to Tempo.

How to use it

1. Turn on exemplar storage in Prometheus.
2. Map the trace destination on the data source.
3. Toggle Exemplars on the histogram query.
4. Click a diamond to open the trace in Tempo.

Useful commands

# Confirm Prometheus has exemplar storage enabled
curl -s http://prometheus:9090/api/v1/status/flags | jq '.data["enable-feature"]'

# Query exemplars directly from Prometheus
curl -s http://prometheus:9090/api/v1/query_exemplars \
  --data-urlencode 'query=http_request_duration_seconds_bucket' \
  --data-urlencode 'start=2026-07-03T00:00:00Z' \
  --data-urlencode 'end=2026-07-03T01:00:00Z' | jq '.data | length'

# Inspect the data source exemplar destination mapping
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://grafana:3000/api/datasources/name/Prometheus | jq '.jsonData.exemplarTraceIdDestinations'

Example config

# provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom-uid
    url: http://prometheus:9090
    jsonData:
      exemplarTraceIdDestinations:
        - name: trace_id
          datasourceUid: tempo-uid
          urlDisplayLabel: 'View Trace in Tempo'

# Prometheus started with exemplar storage (systemd/args)
# --enable-feature=exemplar-storage
# storage.exemplars.exemplars-limit tuned in prometheus.yml if needed

Common findings this catches

• No diamonds on panel → exemplar storage feature not enabled.
• Diamonds but no trace link → exemplarTraceIdDestinations missing.
• Wrong trace opens → label name mismatch (trace_id vs traceID).
• Exemplars disappear → remote-write dropping them.
• Gauge has no exemplars → wrong metric type.
• Exemplars sparse → in-memory cap evicting them.

When to escalate

• Client library upgrade needed for exemplar support — dev team.
• Mimir remote-write exemplar forwarding at scale — platform.
• Sampling strategy so the linked trace still exists in Tempo — architecture.