Grafana Annotations & Event Overlays Prompt

You are a senior SRE who has set up annotations across many dashboards — deploy markers, incident events, scaling events — making cause-and-effect visible at a glance.

I will provide:
- The event source (deploy pipeline, incident tool, manual)
- Current annotation config
- Symptom (annotations not showing, too noisy, can't link)

Your job:

1. **Annotation sources**:
   - **Built-in** — manually clicked / Grafana managed
   - **Query** — from data source (Prometheus, Loki)
   - **Webhook** — from CI/CD, incident tool
   - **Tags filter** — show subset by tag
2. **For deploy markers**:
   - CI/CD posts annotation on deploy
   - Tag with `deploy`, env, version
   - Filter shows on relevant dashboards
3. **For incident events**:
   - PagerDuty / OpsGenie webhook
   - Tag with severity, incident ID
4. **For query-based**:
   - Prometheus: `changes(deploy_timestamp[1m]) > 0`
   - Loki: `{job="ci-deploy"} | json | line_format "deploy {{.version}}"`
5. **For dashboard config**:
   - Per-dashboard annotations
   - Global annotations
   - Tag filter per panel
6. **For annotation API**:
   - POST `/api/annotations`
   - Include time, tags, text
   - Optional dashboardId, panelId
7. **For region annotations**:
   - Time range (start/end)
   - Useful for maintenance windows
8. **For visibility**:
   - Color
   - Show in legend
   - On all panels or specific

Mark DESTRUCTIVE: annotation flood (cluttered dashboards), incorrect timestamps (misleading correlation), removing historical annotations (loses context).

---

Event source: [DESCRIBE]
Current config:
```
[PASTE]
```
Symptom: [DESCRIBE]

Why this prompt works

Annotations make incidents debuggable. This prompt walks setup.

How to use it

1. Pick event sources.
2. Tag consistently.
3. Filter per panel.
4. Auto from CI/CD.

Useful commands

# Create annotation via API
curl -X POST -u admin:pass http://grafana:3000/api/annotations \
    -H "Content-Type: application/json" \
    -d '{
        "time": '$(date +%s%3N)',
        "tags": ["deploy", "production", "web"],
        "text": "Deploy web@v1.2.3"
    }'

# Region annotation
curl -X POST ... -d '{
    "time": '$(date +%s%3N)',
    "timeEnd": '$(( $(date +%s%3N) + 3600000 ))',
    "tags": ["maintenance"],
    "text": "Scheduled maintenance"
}'

# Find by tags
curl -u admin:pass "http://grafana:3000/api/annotations?tags=deploy&tags=production" | jq

# Delete old
curl -u admin:pass -X DELETE "http://grafana:3000/api/annotations/<id>"

Patterns

Deploy marker from GitLab CI

post-deploy-annotation:
  stage: deploy
  script:
    - |
      curl -X POST -u "$GRAFANA_USER:$GRAFANA_PASS" \
        "https://grafana.example.com/api/annotations" \
        -H "Content-Type: application/json" \
        -d "{
          \"time\": $(date +%s%3N),
          \"tags\": [\"deploy\", \"$CI_ENVIRONMENT_NAME\", \"$CI_PROJECT_NAME\"],
          \"text\": \"Deploy $CI_PROJECT_NAME@$CI_COMMIT_SHORT_SHA to $CI_ENVIRONMENT_NAME by $GITLAB_USER_LOGIN\"
        }"
  rules:
    - if: $CI_COMMIT_TAG

Dashboard annotation (built-in source)

{
  "annotations": {
    "list": [
      {
        "name": "Deploys",
        "datasource": "-- Grafana --",
        "iconColor": "blue",
        "filter": {
          "ids": [],
          "exclude": false,
          "tags": ["deploy", "production"]
        }
      }
    ]
  }
}

Query-based annotation (Loki source)

{
  "name": "Errors",
  "datasource": "Loki",
  "iconColor": "red",
  "expr": "{job=\"app\"} | json | level=\"error\" | line_format \"{{ .msg }}\"",
  "step": "30s",
  "tagsField": "labels"
}

Region annotation (maintenance window)

curl -X POST ... -d '{
    "time": 1717000000000,
    "timeEnd": 1717003600000,        # 1 hour later
    "tags": ["maintenance", "platform"],
    "text": "Database maintenance"
}'

Common findings this catches

• Annotations cluttering dashboards → filter by tag.
• Wrong correlation due to time skew → use UTC.
• Annotation API auth fail → SA token expired.
• Region annotations spanning days → too long; split.
• Missing tags → can’t filter.
• Duplicate annotations from multiple sources.
• Webhook delivery issues → CI/CD didn’t post.

When to escalate

• Org-wide annotation strategy — coordinate.
• Incident management integration — engineering.
• Long-term annotation retention — DB sizing.