Grafana Data Source Provisioning YAML Prompt

You are a senior platform engineer who provisions Grafana data sources declaratively from /etc/grafana/provisioning/datasources so nothing is clicked in the UI.

I will provide:
- The data source types (Prometheus, Loki, Tempo, CloudWatch, SQL)
- The endpoints, auth model, and secret source
- Whether the source should be default or read-only

Your job:

1. **File layout**: place YAML under `/etc/grafana/provisioning/datasources/*.yaml` with `apiVersion: 1` and a `datasources:` list; Grafana loads it on boot.
2. **Core fields**: set `name`, `type` (e.g. `prometheus`, `loki`, `tempo`, `cloudwatch`, `postgres`), `uid` (stable, for dashboard references), `url`, `access: proxy`, and `orgId`.
3. **Secrets**: never inline tokens — use `secureJsonData` with `${ENV_VAR}` expansion (enable `[security] disable_initial_admin_creation`-style env interpolation) and keep passwords out of `jsonData`.
4. **Per-type jsonData**: Prometheus `httpMethod: POST`, `prometheusType`, `timeInterval`; Loki `maxLines`; Tempo `tracesToLogsV2`; CloudWatch `authType`, `defaultRegion`; SQL `sslmode`, `maxOpenConns`.
5. **Correlations**: wire trace/log/metric linking via `derivedFields` (Loki), `tracesToLogs`/`tracesToMetrics` (Tempo), and `exemplarTraceIdDestinations` (Prometheus).
6. **Lifecycle**: use `deleteDatasources:` to remove stale ones and set `editable: false` so the UI can't drift from code.
7. **Idempotency**: `uid` makes re-provisioning update-in-place; changing `uid` breaks dashboard references.
8. **Validate**: check container logs on restart for provisioning errors and test each source's health endpoint.

Mark DESTRUCTIVE: `deleteDatasources` removing a source dashboards depend on, changing a live `uid` (breaks every panel), or pointing a name at the wrong cluster.

---

Data sources needed: [DESCRIBE]
Endpoints/auth: [DESCRIBE]
Secret source: [DESCRIBE]

Why this prompt works

Data source provisioning is the foundation of config-as-code Grafana, but the field names differ per type and secrets handling is easy to get wrong. This prompt pins the file layout, forces secureJsonData with env expansion, and makes uid stability explicit so dashboards keep resolving after a redeploy.

How to use it

1. List every data source type so per-type jsonData is correct.
2. Name the secret source (env, Vault-injected env) to keep tokens out of YAML.
3. Fix uid values early and never change them.
4. Restart and check logs to confirm provisioning succeeded.

Useful commands

# Validate provisioning loaded (check startup logs)
grafana-cli admin data-migration encrypt-datasource-passwords   # migrate legacy secrets
journalctl -u grafana-server | grep -i provisioning

# List data sources via API
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/datasources | jq '.[] | {name,uid,type}'

# Test a data source health
curl -s -H "Authorization: Bearer $TOKEN" \
  http://localhost:3000/api/datasources/uid/prometheus-prod/health | jq

Relevant grafana.ini:

[paths]
provisioning = /etc/grafana/provisioning

Example config

/etc/grafana/provisioning/datasources/core.yaml:

apiVersion: 1
deleteDatasources:
  - name: OldProm
    orgId: 1
datasources:
  - name: Prometheus
    uid: prometheus-prod
    type: prometheus
    access: proxy
    url: http://mimir-query-frontend:9090/prometheus
    isDefault: true
    editable: false
    jsonData:
      httpMethod: POST
      prometheusType: Mimir
      timeInterval: 30s
      exemplarTraceIdDestinations:
        - name: traceID
          datasourceUid: tempo-prod
  - name: Loki
    uid: loki-prod
    type: loki
    access: proxy
    url: http://loki-gateway:3100
    editable: false
    jsonData:
      maxLines: 1000
      derivedFields:
        - name: TraceID
          matcherRegex: 'traceID=(\w+)'
          url: '$${__value.raw}'
          datasourceUid: tempo-prod
  - name: CloudWatch
    uid: cw-prod
    type: cloudwatch
    editable: false
    jsonData:
      authType: default
      defaultRegion: us-east-1
    secureJsonData:
      accessKey: ${CW_ACCESS_KEY}
      secretKey: ${CW_SECRET_KEY}

Common findings this catches

• Blank panels after redeploy → uid changed or source deleted.
• Secrets in git → tokens inlined instead of secureJsonData.
• UI drift → editable: true allowed manual edits.
• No trace/log linking → missing derivedFields/exemplar config.
• Grafana won’t start → YAML syntax error in provisioning.
• Wrong region/db → empty results mistaken for outage.

When to escalate

• Rotating credentials across many sources — secrets management owner.
• Multi-org / multi-tenant data source strategy — platform team.
• Cross-source correlation design (traces↔logs↔metrics) — observability lead.