Grafana Alert Rules Provisioning YAML Prompt

You are a senior observability engineer who provisions Grafana unified alerting entirely as code from /etc/grafana/provisioning/alerting so nothing is created by clicking in the UI.

I will provide:
- The signals to alert on (PromQL/LogQL expressions and datasource UIDs)
- Thresholds, `for` durations, and severity labels
- The folder/group layout and how alerts should route to contact points

Your job:

1. **File layout**: emit YAML under `/etc/grafana/provisioning/alerting/*.yaml` with `apiVersion: 1` and top-level `groups:`, `contactPoints:`, `policies:`, and optionally `muteTimes:` — Grafana loads these at boot.
2. **Rule groups**: each group needs `orgId`, `name`, `folder`, `interval` (evaluation cadence), and a `rules:` list. Keep `interval` no shorter than the query cost allows.
3. **Rule anatomy**: for every rule set a stable `uid`, `title`, `condition` (the ref id that decides firing), and a `data:` array of query stages — the datasource query (`refId: A`), a reduce/math expression (`refId: B` using datasource `__expr__`), and a threshold (`refId: C`).
4. **Expressions**: use the server-side expression datasource (`type: __expr__`) for `reduce` (last/mean), `math`, and `threshold` stages so alerting evaluates a single scalar per series, avoiding "wide series"/"multi-dimensional" errors.
5. **Timing & no-data**: set `for` (pending duration), `noDataState` (`NoData`/`OK`/`Alerting`), and `execErrState` deliberately — flapping and false pages usually come from these.
6. **Labels & annotations**: attach `labels` (e.g. `severity`, `team`) that notification `policies` match on, and `annotations` (`summary`, `description`, `runbook_url`) using `{{ $values }}`/`{{ $labels }}` templating.
7. **Routing**: define `contactPoints:` and a `policies:` tree that matches labels to receivers, with `group_by`, `group_wait`, `group_interval`, and `repeat_interval`.
8. **Validate**: confirm rules loaded via the Alerting API and check startup logs for provisioning errors; provisioned rules are read-only in the UI (`provenance`).

Mark DESTRUCTIVE: changing a rule `uid` (orphans state/silences), lowering `for` to 0 (page storms), overwriting a hand-tuned group whose YAML you don't have, or a policy change that silently reroutes pages away from on-call.

---

Signals + datasource UIDs: [DESCRIBE]
Thresholds / for / severity: [DESCRIBE]
Folder + routing layout: [DESCRIBE]

Why this prompt works

Grafana’s unified alerting model is powerful but has sharp edges: rules are multi-stage query pipelines, the server-side expression datasource is mandatory for real thresholds, and provisioning YAML has its own schema distinct from the legacy alert blocks embedded in dashboards. This prompt pins the file layout, forces the reduce/math/threshold expression chain that avoids “wide series” evaluation errors, and makes uid stability and routing explicit so pages actually reach on-call.

How to use it

1. Give exact PromQL/LogQL plus datasource uids so the query stage is correct.
2. State the for and severity for every signal to control flapping and routing.
3. Describe the routing tree (which labels map to which contact point).
4. Provision, then verify via the API before trusting the rules in production.

Useful commands

# Reload provisioning without a full restart (Grafana 9.5+)
curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/admin/provisioning/alerting/reload

# List provisioned alert rules
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[] | {uid,title,folderUID}'

# Watch startup / provisioning errors
journalctl -u grafana-server | grep -iE "provisioning|alerting|ngalert"

Relevant grafana.ini:

[unified_alerting]
enabled = true

[paths]
provisioning = /etc/grafana/provisioning

Example config

/etc/grafana/provisioning/alerting/api-latency.yaml:

apiVersion: 1
groups:
  - orgId: 1
    name: api-slo
    folder: SLOs
    interval: 1m
    rules:
      - uid: api-p95-latency
        title: API p95 latency high
        condition: C
        for: 5m
        noDataState: NoData
        execErrState: Error
        labels:
          severity: page
          team: platform
        annotations:
          summary: "p95 latency {{ $values.B }} over threshold"
          runbook_url: "https://runbooks.example.com/api-latency"
        data:
          - refId: A
            datasourceUid: prometheus-prod
            relativeTimeRange: { from: 600, to: 0 }
            model:
              expr: job:http_request_duration_seconds:p95
              instant: false
          - refId: B
            datasourceUid: __expr__
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator: { type: gt, params: [0.5] }
contactPoints:
  - orgId: 1
    name: platform-pager
    receivers:
      - uid: platform-pd
        type: pagerduty
        settings:
          integrationKey: ${PD_ROUTING_KEY}
policies:
  - orgId: 1
    receiver: platform-pager
    group_by: [alertname, team]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: platform-pager
        matchers: ["severity = page"]

Common findings this catches

• “Input data must be a wide series” on eval → missing reduce stage before the threshold.
• Page storms → for too short or absent.
• Pages never arrive → policy matcher doesn’t match the rule’s labels.
• Silent broken alerts → execErrState: OK masking query errors.
• Duplicate/ghost alerts → a changed uid re-registered the rule.
• Grafana won’t start → YAML error in an alerting provisioning file.

When to escalate

• Cross-team routing and escalation policy design — on-call/incident owner.
• SLO burn-rate and multi-window alerting math — reliability lead.
• Migrating legacy dashboard alerts to unified alerting at scale — platform team.