Grafana PagerDuty/Opsgenie Contact Point Prompt

You are a senior SRE who has integrated Grafana Alerting with PagerDuty and Opsgenie for production on-call.

I will provide:
- The paging tool (PagerDuty and/or Opsgenie) and integration keys
- Alert rules and their labels (severity, team, service)
- Routing and escalation requirements

Your job:

1. **Create contact points**: a PagerDuty contact point using the Events API v2 integration/routing key, and an Opsgenie contact point using the API key and region (US/EU).
2. **Map severity**: set PagerDuty `severity` (critical/error/warning/info) and Opsgenie `priority` (P1-P5) from alert labels via templating.
3. **Dedup keys**: use a stable `dedup_key`/`alias` (e.g. `{{ .GroupLabels.alertname }}-{{ .GroupLabels.service }}`) so re-fires and resolves group correctly.
4. **Notification policy tree**: route by label matchers (`team=payments`, `severity=critical`) to the right contact point with nested policies.
5. **Grouping and timing**: set `group_by`, `group_wait`, `group_interval`, and `repeat_interval` to balance noise vs latency.
6. **Mute timings**: attach maintenance-window mute timings to non-urgent routes.
7. **Resolve behavior**: ensure resolved alerts send resolve events so incidents auto-close.
8. **Provisioning**: express contact points and policies as Alerting provisioning YAML with secrets externalized.

Mark DESTRUCTIVE: changing the default policy (affects all alerts), rotating an integration key in place (silences a live route), overly broad matchers that swallow other teams' alerts.

---

Paging tool/keys: [DESCRIBE]
Alert rules/labels: [DESCRIBE]
Routing/escalation: [DESCRIBE]

Why this prompt works

Paging integrations fail in subtle ways — a wrong region, an unstable dedup key, or a too-broad matcher that reroutes another team’s alerts. This prompt structures the contact point, severity mapping, dedup strategy, and the notification-policy tree together, which is the only way routing behaves predictably under load.

How to use it

1. Provide the integration keys and region so the assistant builds the correct contact point.
2. List alert labels so it maps severity/priority and writes matchers.
3. State escalation intent so grouping and repeat intervals fit your SLA.
4. Ask for provisioning YAML with secrets externalized.

Useful commands

# Test a contact point via the Grafana Alerting API
curl -X POST http://localhost:3000/api/v1/provisioning/contact-points/test \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"pd-payments","type":"pagerduty"}'

# Export current notification policy tree
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/v1/provisioning/policies | jq .

Example config

# provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: pd-payments
    receivers:
      - uid: pd-payments-1
        type: pagerduty
        settings:
          integrationKey: ${PD_PAYMENTS_KEY}
          severity: '{{ .CommonLabels.severity }}'
          dedup_key: '{{ .GroupLabels.alertname }}-{{ .GroupLabels.service }}'
  - orgId: 1
    name: og-platform
    receivers:
      - uid: og-platform-1
        type: opsgenie
        settings:
          apiKey: ${OPSGENIE_KEY}
          apiUrl: https://api.eu.opsgenie.com/v2/alerts
          priority: '{{ if eq .CommonLabels.severity "critical" }}P1{{ else }}P3{{ end }}'
---
# provisioning/alerting/policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: grafana-default-email
    group_by: ['alertname', 'service']
    routes:
      - receiver: pd-payments
        object_matchers: [['team', '=', 'payments'], ['severity', '=', 'critical']]
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 4h

Common findings this catches

• Wrong on-call paged → over-broad label matcher.
• Duplicate incidents → unstable dedup key/alias.
• Dropped pages → wrong Opsgenie region URL.
• Alert storms → repeat_interval too short.
• Stuck open incidents → resolve events not sent.
• Secret exposure → integration key inline instead of externalized.
• Missed criticals → severity not mapped from labels.

When to escalate

• On-call schedule/escalation design — coordinate with the paging tool admin.
• Repeated missed pages during incidents — treat as a reliability incident itself.
• Multi-team routing governance — platform/alerting standards owner.