Grafana Alertmanager Datasource & Routing Prompt

You are a senior SRE who configures Grafana's Alertmanager integration and writes routing trees that group and dedupe alerts so on-call gets signal, not noise.

I will provide:
- Whether alerts come from Grafana-managed rules, an external Prometheus Alertmanager, or both
- The receivers available (PagerDuty, Slack, email, webhook) and which teams own which alerts
- The label schema on alerts (severity, team, cluster, namespace)

Your job:

1. **Datasource / mode**: decide whether to use the built-in Grafana Alertmanager or add an external one as an `alertmanager` datasource (`implementation: prometheus` or `mimir`), and provision it as YAML with the correct `handleGrafanaManagedAlerts` setting.
2. **Routing tree**: build a `route` with a sane root (`group_by`, `group_wait`, `group_interval`, `repeat_interval`) and child routes that match on labels (`team`, `severity`) with `continue` used deliberately.
3. **Grouping**: choose `group_by` labels so related alerts collapse into one notification (e.g. by `cluster` + `alertname`) instead of one message per series.
4. **Inhibition**: write `inhibit_rules` so a `severity: critical` cluster-down alert suppresses the downstream `warning` noise it causes.
5. **Receivers**: map each route to a receiver with the right integration config, and reference secrets, never inline them.
6. **Silences & mute timings**: define `mute_time_intervals` for maintenance windows and explain silences vs mutes.
7. **Validation**: show how to `amtool` check the config and test-route a sample alert before shipping.

Mark DESTRUCTIVE: none — but a wrong root `group_by` or `repeat_interval` can either flood on-call or hide real pages.

---

Alert sources: [DESCRIBE]
Receivers + team ownership: [DESCRIBE]
Alert label schema: [DESCRIBE]

Why this prompt works

Alert routing failures are rarely about the alert rule — they’re about grouping, repeat intervals, and inhibition in the Alertmanager config. This prompt makes the assistant reason about the label schema first, then build a routing tree with deliberate grouping and inhibition, and validate it with amtool before it ever pages a human.

How to use it

1. State whether alerts are Grafana-managed, external, or both so the datasource mode is right.
2. Give the label schema so group_by and route matchers are correct.
3. Map receivers to teams so each route lands on the right on-call.
4. Always validate with amtool against real sample alerts.

Useful commands

# Validate the Alertmanager config and simulate routing
amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml \
  severity=critical team=payments

# Confirm the external Alertmanager datasource is reachable from Grafana
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/datasources/name/Alertmanager | jq '.jsonData.implementation'

Example config

route:
  receiver: default-slack
  group_by: ['cluster', 'alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['team = payments', 'severity = critical']
      receiver: payments-pagerduty
      continue: false
inhibit_rules:
  - source_matchers: ['severity = critical', 'alertname = ClusterDown']
    target_matchers: ['severity = warning']
    equal: ['cluster']
mute_time_intervals:
  - name: weekend-maintenance
    time_intervals:
      - weekdays: ['saturday', 'sunday']

Common findings this catches

• On-call flooded → group_by too narrow or short repeat_interval.
• Missed pages hidden in noise → group_by too broad.
• Double notifications → misused continue: true.
• Suppressed real alerts → inhibition equal labels wrong.
• Leaked secrets → integration keys inlined in the config.

When to escalate

• On-call tolerance and paging policy — the incident-management owner.
• External Alertmanager HA/clustering — the platform team.
• Team↔receiver ownership disputes — the alerting governance owner.