Grafana Provisioning as Code Prompt

You are a senior platform engineer who has provisioned Grafana via code — data sources, dashboards, alerts — and used the sidecar pattern in Kubernetes.

I will provide:
- The use case (new dashboard, multi-env Grafana, GitOps)
- Current state (UI-managed vs file-provisioned)
- Goal

Your job:

1. **Provisioning sources**:
   - **File provisioning** — YAML configs in /etc/grafana/provisioning
   - **API** — REST endpoints
   - **Operator** — Grafana Operator
   - **Terraform provider** — `grafana/grafana`
2. **For data sources**:
   - YAML in `/etc/grafana/provisioning/datasources/`
   - Auth via API key or service account token
   - Multiple DSes; default flag
3. **For dashboards**:
   - YAML in `/etc/grafana/provisioning/dashboards/`
   - References JSON files in folder
   - Auto-reload on file change
4. **For alerts**:
   - YAML in `/etc/grafana/provisioning/alerting/`
   - Rules, contact points, policies, mute timings
5. **For sidecar pattern (Kubernetes)**:
   - kiwigrid/k8s-sidecar reads ConfigMaps labeled `grafana_dashboard=1`
   - Auto-loads dashboards from ConfigMaps
   - Useful for app-distributed dashboards
6. **For multi-env**:
   - Same code, different values per env
   - Use templating (Helm values, Kustomize)
7. **For GitOps**:
   - Dashboards JSON in Git
   - CI updates ConfigMap or pushes via API
   - ArgoCD/Flux deploys
8. **For Jsonnet / libsonnet**:
   - Programmatic dashboard generation
   - Mixin pattern (e.g., kubernetes-mixin)

Mark DESTRUCTIVE: overwriting UI changes with provisioned (UI edits lost), removing data source while dashboards depend, malformed YAML breaking provision.

---

Use case: [DESCRIBE]
Current state: [DESCRIBE]
Goal: [DESCRIBE]

Why this prompt works

Provisioning enables scale + reproducibility. This prompt walks options.

How to use it

1. Pick mechanism based on team comfort.
2. File provisioning for simple.
3. Sidecar for app-distributed.
4. Terraform / Operator for full IaC.

Useful commands

# File provisioning structure (Grafana)
/etc/grafana/provisioning/
├── datasources/
│   └── prometheus.yaml
├── dashboards/
│   └── default.yaml
├── alerting/
│   └── rules.yaml
└── plugins/

# Reload (Grafana 9+)
curl -X POST -u admin:pass http://grafana:3000/api/admin/provisioning/datasources/reload
curl -X POST -u admin:pass http://grafana:3000/api/admin/provisioning/dashboards/reload
curl -X POST -u admin:pass http://grafana:3000/api/admin/provisioning/alerting/reload

# Sidecar pod (k8s-sidecar)
kubectl get pods -l app=grafana
# Check the sidecar container logs

# Dashboard via API
curl -u admin:pass -X POST -H "Content-Type: application/json" \
    -d @dashboard.json \
    http://grafana:3000/api/dashboards/db

Patterns

Data source provisioning

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
- name: Prometheus
  type: prometheus
  url: http://prometheus:9090
  access: proxy
  isDefault: true
- name: Prometheus-Long
  type: prometheus
  url: http://thanos-querier:9090
  access: proxy
- name: Loki
  type: loki
  url: http://loki:3100
  access: proxy

Dashboard provisioning

# /etc/grafana/provisioning/dashboards/default.yaml
apiVersion: 1
providers:
- name: 'default'
  orgId: 1
  folder: 'SRE'
  type: file
  disableDeletion: false
  updateIntervalSeconds: 30
  options:
    path: /var/lib/grafana/dashboards/sre

Then place JSON files in /var/lib/grafana/dashboards/sre/.

Kubernetes sidecar pattern

# Helm chart values
sidecar:
  dashboards:
    enabled: true
    label: grafana_dashboard
    labelValue: "1"
    folderAnnotation: grafana_folder

# ConfigMap with dashboard JSON
apiVersion: v1
kind: ConfigMap
metadata:
  name: web-dashboard
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_folder: "Web"
data:
  web.json: |
    {
      "title": "Web Service",
      ...
    }

Terraform

resource "grafana_data_source" "prometheus" {
  type = "prometheus"
  name = "Prometheus"
  url  = "http://prometheus:9090"
}

resource "grafana_dashboard" "web" {
  config_json = file("dashboards/web.json")
  folder      = grafana_folder.sre.id
}

resource "grafana_folder" "sre" {
  title = "SRE"
}

resource "grafana_alert_rule" "high_cpu" {
  name      = "HighCPU"
  folder_uid = grafana_folder.sre.uid
  interval_seconds = 60
  data {
    ref_id = "A"
    datasource_uid = grafana_data_source.prometheus.uid
    relative_time_range {
      from = 600
      to = 0
    }
    model = jsonencode({
      expr = "avg(rate(node_cpu_seconds_total[5m])) > 0.8"
    })
  }
}

Jsonnet / Grafonnet

local g = import 'grafana-builder/grafana.libsonnet';
g.dashboard('Service Overview')
.addRow(
  g.row('HTTP')
  .addPanel(
    g.panel('Requests/sec')
    .addTarget(g.target('sum(rate(http_requests_total[5m]))'))
  )
)

Common findings this catches

• Provisioned dashboard can’t be saved from UI → expected; edit in code.
• Sidecar not reloading → label mismatch on ConfigMap.
• Data source URL wrong → connection error per panel.
• Multi-env shared config → use templating per env.
• Dashboard JSON > 1MB → split into separate ConfigMaps.
• API key compromised → rotate; use service account token.
• Provisioning reload fails → YAML syntax; check logs.

When to escalate

• Migration from UI-edited to provisioned — staged.
• Library/mixin adoption — strategic.
• Multi-org Grafana — federation.