Grafana Azure Monitor Data Source Design Prompt

You are a senior observability engineer who has wired Grafana to Azure Monitor across many subscriptions.

I will provide:
- Azure services and subscriptions in scope
- Auth model (managed identity vs app registration)
- Dashboards or SLOs to power

Your job:

1. **Pick the auth path**: app registration (client ID/secret) or managed identity when Grafana runs in Azure; grant `Monitoring Reader` at the right scope.
2. **Metrics service**: use the Metrics query type — namespace, resource, metric, aggregation (Average/Total/Max), and dimension splits; note per-metric supported aggregations.
3. **Logs service (Log Analytics)**: write KQL against a workspace; use `$__timeFilter(TimeGenerated)` and `$__interval` macros; return time series with `| summarize ... by bin(TimeGenerated, $__interval)`.
4. **Azure Resource Graph**: enumerate resources across subscriptions with KQL-style ARG queries for inventory panels.
5. **Template variables**: chain subscription -> resource group -> resource using the datasource query types so dashboards stay dynamic.
6. **Rate limits and cost**: batch metric requests, widen intervals, and avoid per-second polling that trips Azure API throttling.
7. **Provisioning**: define the datasource in YAML with `jsonData` (subscriptionId, tenantId, cloudName) and `secureJsonData` (clientSecret).
8. **Validate**: test each query type, confirm timezone and aggregation, and check dashboard variable refresh.

Mark DESTRUCTIVE: rotating a client secret in place (breaks live datasource), over-broad RBAC role assignment, deleting a datasource referenced by dashboards.

---

Services/subscriptions: [DESCRIBE]
Auth model: [DESCRIBE]
Dashboards/SLOs: [DESCRIBE]

Why this prompt works

Azure Monitor is really three query surfaces behind one Grafana plugin — Metrics, Logs (Log Analytics/KQL), and Resource Graph — each with its own auth scope, macros, and gotchas. Engineers routinely mix up aggregations or leave KQL unbounded. This prompt forces the model to separate the three services, apply the right macros, and lock down auth before anything reaches a dashboard.

How to use it

1. State the auth model — managed identity or app registration — so the assistant picks the correct jsonData shape.
2. List services (VMSS, AKS, App Service, etc.) so it maps metric namespaces.
3. Ask for chained template variables to keep dashboards subscription-agnostic.
4. Request provisioning YAML so the datasource is reproducible.

Useful commands

# Create the app registration and grant least-privilege read
az ad sp create-for-rbac --name grafana-monitor --role "Monitoring Reader" \
  --scopes /subscriptions/<sub-id>

# Test datasource via Grafana HTTP API
curl -s -H "Authorization: Bearer $GRAFANA_TOKEN" \
  http://localhost:3000/api/datasources/uid/azuremon/health

# Provision datasources from file
curl -X POST http://localhost:3000/api/datasources \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  -H "Content-Type: application/json" \
  -d @azure-monitor-ds.json

Example config

# provisioning/datasources/azure-monitor.yaml
apiVersion: 1
datasources:
  - name: Azure Monitor
    type: grafana-azure-monitor-datasource
    uid: azuremon
    access: proxy
    jsonData:
      azureAuthType: clientsecret
      cloudName: azuremonitor
      tenantId: ${AZURE_TENANT_ID}
      clientId: ${AZURE_CLIENT_ID}
      subscriptionId: ${AZURE_SUBSCRIPTION_ID}
    secureJsonData:
      clientSecret: ${AZURE_CLIENT_SECRET}

// Log Analytics (KQL) time series with Grafana macros
AzureMetrics
| where $__timeFilter(TimeGenerated)
| where ResourceProvider == "MICROSOFT.COMPUTE"
| summarize avg(Average) by bin(TimeGenerated, $__interval), Resource
| order by TimeGenerated asc

Common findings this catches

• Wrong aggregation → Total used where Average is meant, inflating panels.
• Unbounded KQL → missing $__timeFilter(TimeGenerated) scans full retention.
• Throttling (429) → polling interval too tight across many resources.
• Broken variable chain → resource group variable not scoped to subscription.
• Secret in jsonData → must move to secureJsonData.
• Empty panels → metric namespace/dimension mismatch.
• Timezone drift → dashboard not aligned to UTC vs local.

When to escalate

• Cross-tenant access design — involve Azure identity/security team.
• Large-scale Resource Graph inventory across hundreds of subscriptions — capacity planning.
• Persistent throttling despite tuning — open a support case with Azure.