Grafana Prometheus Datasource Design Prompt

You are a senior observability engineer who configures Grafana Prometheus datasources so that panels are fast, honest about resolution, and provisioned as code.

I will provide:
- The backend (vanilla Prometheus, Thanos Query, Mimir, or Cortex) and its URL/auth
- The global `scrape_interval` and typical query load
- Whether exemplars, recording rules, and a Tempo/Loki correlation are in play

Your job:

1. **Datasource YAML**: emit a `datasources.yaml` provisioning block with `type: prometheus`, `access: proxy`, the URL, and `jsonData` set correctly — `httpMethod: POST` (so long selectors don't hit URL limits), `prometheusType` and `prometheusVersion` (so the query editor enables the right features), and `timeInterval` set to the real scrape interval so `$__rate_interval` computes sanely.
2. **Resolution honesty**: explain how `timeInterval`, `$__interval`, and `$__rate_interval` interact; recommend `$__rate_interval` for all `rate()`/`irate()` so panels never under-sample when zoomed out.
3. **Timeouts & limits**: set `queryTimeout` and, for Mimir/Thanos, note the server-side `-querier.timeout` and max-samples so a panel failure is diagnosed on the right side.
4. **Exemplars**: if enabled, add `exemplarTraceIdDestinations` linking `traceID` to the Tempo datasource UID for metrics→traces drilldown.
5. **Recording rules**: recommend which heavy panel expressions should be pushed to recording rules, and show the panel query rewritten to read the recorded series.
6. **Auth**: TLS/bearer/sigv4 as needed via `secureJsonData`, never inline secrets.
7. **Alerting**: note `manageAlerts` and whether this datasource is the unified-alerting rule target.

Mark DESTRUCTIVE: none — read-only. But warn that a wrong `timeInterval` silently distorts every rate panel.

---

Backend + URL/auth: [DESCRIBE]
Scrape interval + load: [DESCRIBE]
Exemplars / correlation: [DESCRIBE]

Why this prompt works

The Prometheus datasource looks like a one-line URL field, but three settings — httpMethod, timeInterval, and prometheusType/Version — quietly govern whether panels are correct and fast. This prompt forces them to be set deliberately, aligns $__rate_interval to the real scrape interval, and provisions everything as YAML so the config is reproducible rather than clicked in.

How to use it

1. Name the backend (Prometheus vs Thanos/Mimir/Cortex) so version-specific features are enabled correctly.
2. Give the real scrape interval so timeInterval and rate math are honest.
3. Say whether exemplars and correlation matter so the trace drilldown is wired.
4. Ask for provisioning YAML, not UI steps.

Useful commands

# Validate the provisioned datasource is healthy
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/datasources/name/Prometheus | jq '.jsonData'

# Confirm the backend's own query timeout
curl -s http://prometheus:9090/api/v1/status/flags | jq '."query.timeout"'

Example config

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus-prod
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      prometheusType: Mimir
      prometheusVersion: 2.14.0
      timeInterval: 15s
      queryTimeout: 60s
      manageAlerts: true
      exemplarTraceIdDestinations:
        - name: traceID
          datasourceUid: tempo-prod
    secureJsonData:
      httpHeaderValue1: "Bearer ${PROM_TOKEN}"

Common findings this catches

• Wrong rate values → timeInterval not matching scrape_interval.
• Truncated queries → httpMethod: GET on long selectors.
• Missing exemplar button → prometheusType/Version unset or wrong.
• Slow panels blamed on Grafana → real limit is server-side -querier.timeout.
• Dead trace links → exemplarTraceIdDestinations pointing at wrong UID.

When to escalate

• Server-side query limits/timeouts (Mimir/Thanos) — the metrics platform team.
• Recording-rule strategy and cardinality — the Prometheus rule owner.
• Cross-datasource correlation design — the observability lead.