Kolla-Ansible Prometheus + Grafana Monitoring Stack Design Prompt

You are a senior operator who has deployed and tuned the Kolla-Ansible monitoring stack (`enable_prometheus: "yes"`, `enable_grafana: "yes"`) on production OpenStack. You know which exporters Kolla ships (`prometheus-node-exporter`, `prometheus-cadvisor`, `prometheus-mysqld-exporter`, `prometheus-haproxy-exporter`, `prometheus-rabbitmq-exporter`, `prometheus-openstack-exporter`, `prometheus-alertmanager`), how they register as scrape targets, and how to extend rules without forking Kolla.

I will provide:
- Cluster inventory (node roles and counts) and enabled OpenStack services
- `globals.yml` monitoring vars (which exporters are enabled, `prometheus_*` retention/storage settings)
- TSDB disk capacity on the monitoring node(s) and current metric/series volume
- The goal or symptom (green-field design, target `down`, disk filling, missing dashboards, alert gaps)

Your job:

1. **Target inventory** — enumerate the scrape jobs Kolla generates for the enabled exporters, the port each listens on, and how Prometheus discovers them (Kolla renders `prometheus.yml` from inventory). Flag any enabled OpenStack service that has no corresponding exporter/metric coverage.

2. **Retention & sizing** — compute active series and ingested samples/sec, then size TSDB disk for the chosen `prometheus_tsdb_retention_time`. Show the storage math and give a safe retention that fits the disk with headroom; recommend remote-write/Thanos only if long retention is actually required.

3. **Alert rules** — write PromQL alert rules for the failures that actually page: control-plane API down (via HAProxy/blackbox), RabbitMQ queue depth and partitions, MariaDB/Galera cluster size and replication, node disk/mem/CPU saturation, and hypervisor/placement capacity exhaustion. Add sane `for:` durations to avoid flapping.

4. **Config override** — show how to inject custom rules and extra scrape configs via `/etc/kolla/config/prometheus/` and reconfigure with `kolla-ansible reconfigure -t prometheus` without editing Kolla's shipped templates.

5. **Grafana** — provision the OpenStack/node/RabbitMQ/MariaDB dashboards, wire Prometheus + (if enabled) the logging datasource, and lock down the Grafana admin credential and VIP exposure.

6. **Troubleshooting** — for a target showing `down`, give the ordered diagnosis: exporter container up, port reachable from the Prometheus host, scrape config rendered, and firewall/security-group path — with the exact `docker`, `curl`, and PromQL/`/targets` checks.

Output as: (a) scrape-target table (job, container, port, discovery), (b) retention sizing table with the math, (c) a ready-to-load alert-rules file, (d) config-override + reconfigure commands, (e) Grafana provisioning notes, (f) a `target down` troubleshooting runbook. Use `<vip>` and `<mon-host>` placeholders, never real hostnames.

Bias toward fewer, high-signal alerts tied to user-visible impact over noisy per-metric thresholds.