Grafana Annotations & Region Events Prompt

You are a senior observability engineer who uses Grafana annotations to overlay deploys, incidents, and maintenance windows on metric panels so responders see cause next to effect.

I will provide:
- The events I want to overlay (deploys, feature flags, incidents, maintenance) and where they come from
- The dashboards/panels that should show them
- Whether events are point-in-time or spans (regions with start/end)

Your job:

1. **Annotation source choice**: decide per event type between native annotations (posted via the API/UI), query-driven annotations (from a Prometheus/Loki/SQL datasource), or the built-in `Annotations & Alerts` dashboard annotation.
2. **Region vs point**: for anything with a duration (a deploy rollout, a maintenance window), post a region annotation with `timeEnd` so the overlay is a shaded band, not a single line.
3. **Tags & filtering**: assign consistent tags (`deploy`, `incident`, `env:prod`, `service:checkout`) so a dashboard annotation query can filter by tag, and drive that filter from a template variable.
4. **Automation**: show how a CI pipeline posts a deploy annotation via the HTTP API on every release, and how an alert can auto-create annotations.
5. **Provisioning**: for query-driven annotations, deliver the dashboard JSON `annotations` block wired to the datasource and tag filter.
6. **Hygiene**: recommend a retention/cleanup approach so old annotations don't clutter, and note org-vs-dashboard scope.

Mark DESTRUCTIVE: deleting annotations via the API is irreversible — scope deletes by tag and time carefully.

---

Events to overlay + sources: [DESCRIBE]
Dashboards/panels: [DESCRIBE]
Point-in-time or regions?: [DESCRIBE]

Why this prompt works

Annotations turn “latency spiked at 14:03” into “latency spiked at 14:03, right after the checkout deploy.” The failure modes are subtle: point annotations where regions belong, inconsistent tags that make filters miss events, and unscoped delete tokens. This prompt picks the right source per event, uses regions for durations, standardises tags, and automates deploy annotations from CI.

How to use it

1. List event types and their sources so each maps to native vs query-driven.
2. Say which events are spans so regions get a timeEnd.
3. Agree a tag vocabulary so annotation queries filter reliably.
4. Wire CI to post deploy annotations so the overlay is automatic.

Useful commands

# Post a deploy region annotation from CI
curl -X POST http://localhost:3000/api/annotations \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"time":1720000000000,"timeEnd":1720000180000,
       "tags":["deploy","service:checkout","env:prod"],
       "text":"Deploy v2.14.0"}'

# List annotations by tag before any cleanup
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  "http://localhost:3000/api/annotations?tags=deploy&from=1719000000000" | jq length

Example config

Dashboard annotation block (query-driven, tag-filtered):

"annotations": {
  "list": [
    {
      "name": "Deploys",
      "datasource": { "type": "grafana", "uid": "-- Grafana --" },
      "enable": true,
      "iconColor": "blue",
      "type": "tags",
      "tags": ["deploy", "service:$service"]
    }
  ]
}

Common findings this catches

• Lost rollout windows → point annotations instead of regions.
• Events missing from panels → inconsistent or misspelled tags.
• Cluttered/slow panels → high-cardinality query annotations.
• Accidental mass deletion → broad-scope API token.
• Filter mismatches → org vs dashboard annotation scope confusion.

When to escalate

• Deploy-event pipeline integration — the CI/CD owner.
• Annotation retention policy — the Grafana platform owner.
• Cross-team tag vocabulary — the observability lead.