Grafana Chained Template Variables Prompt

You are a senior observability engineer who designs Grafana dashboard template variables that chain cleanly so one dashboard serves every environment.

I will provide:
- The datasource type (Prometheus, Loki, SQL, CloudWatch)
- The dimensions to filter on and their hierarchy (e.g. region → cluster → namespace → pod)
- Whether the dashboard is single- or multi-select and the default scope

Your job:

1. **Variable hierarchy**: define an ordered set of variables where each `query` filters on the value of the one above using `$upper` interpolation, so selecting a cluster narrows the namespace list, etc.
2. **Variable types**: choose correctly among `query` (datasource-driven), `custom` (static list), `interval` (`$__interval`/`$__rate_interval` tuning), `datasource` (multi-cluster dashboards), `constant`, and `textbox`.
3. **Prometheus queries**: use `label_values(metric{filter=~"$parent"}, label)` — always scope by a metric and the parent variable so the list is small and relevant, not `label_values(label)` across everything.
4. **Multi-select & All**: set `includeAll`, a correct `allValue` (often `.*` for Prometheus regex, empty for exact-match datasources), and `multi` intentionally; wire panel queries with `=~"$var"` so regex/All works.
5. **Interpolation format**: pick the right `${var:regex}`, `${var:pipe}`, `${var:csv}`, or `${var:sqlstring}` format so the value is valid in the target query language.
6. **Performance**: set `refresh` to `On dashboard load` vs `On time range change`, and add a `regex` extract/filter to trim noisy values; avoid re-querying huge label sets on every render.
7. **Sorting & dedup**: enable numeric/alphabetical `sort` and use `regex` capture groups to clean up display values.
8. **Ad hoc filters**: when appropriate, add an `adhoc` variable for free-form label filtering, and note it applies globally to matching datasources.

Explain each choice, then output the dashboard `templating.list` JSON and one example panel query that consumes the variables.

---

Datasource: [DESCRIBE]
Dimensions + hierarchy: [DESCRIBE]
Single/multi-select + defaults: [DESCRIBE]

Why this prompt works

Chained variables are where reusable dashboards live or die: get the interpolation format or the =~"$var" matcher wrong and every panel returns “templating failed to load” or empty data. This prompt forces parent-scoped label_values queries, deliberate includeAll/allValue/multi choices, and the correct ${var:format} per query language so cascading dropdowns stay fast and correct.

How to use it

1. State the exact hierarchy so each child query is scoped by its parent.
2. Name the datasource so interpolation format and query syntax are right.
3. Decide single vs multi-select up front — it changes allValue and the panel matcher.
4. Test the “All” path explicitly; it’s the most common silent bug.

Useful commands

# Preview what a Prometheus variable query returns
curl -s -G "http://prometheus:9090/api/v1/label/namespace/values" \
  --data-urlencode 'match[]=kube_pod_info{cluster="prod-eu"}' | jq '.data | length'

# Export a dashboard to inspect templating.list
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/dashboards/uid/$UID | jq '.dashboard.templating.list'

Example config

templating.list for a region → cluster → namespace chain:

{
  "templating": {
    "list": [
      {
        "name": "region",
        "type": "query",
        "datasource": { "uid": "prometheus-prod" },
        "query": "label_values(kube_pod_info, region)",
        "refresh": 1,
        "sort": 1
      },
      {
        "name": "cluster",
        "type": "query",
        "datasource": { "uid": "prometheus-prod" },
        "query": "label_values(kube_pod_info{region=~\"$region\"}, cluster)",
        "refresh": 2,
        "includeAll": true,
        "multi": true,
        "allValue": ".*"
      },
      {
        "name": "namespace",
        "type": "query",
        "datasource": { "uid": "prometheus-prod" },
        "query": "label_values(kube_pod_info{cluster=~\"$cluster\"}, namespace)",
        "refresh": 2,
        "includeAll": true,
        "multi": true,
        "allValue": ".*"
      }
    ]
  }
}

Example panel query consuming the chain:

sum by (namespace) (
  rate(container_cpu_usage_seconds_total{cluster=~"$cluster", namespace=~"$namespace"}[$__rate_interval])
)

Common findings this catches

• Empty “All” → wrong allValue, or = instead of =~ in the panel.
• Slow dropdowns → unscoped label_values over high-cardinality labels.
• “Templating failed to load values” → child query references a parent that returned nothing, or wrong interpolation format.
• Stale options → refresh: never.
• Duplicate/ugly values → missing regex cleanup and sort.

When to escalate

• Very high-cardinality label design — metrics/cardinality owner.
• Cross-datasource variable strategy (mixed Prometheus/SQL) — platform team.
• Standardizing a variable convention across many dashboards — observability lead.