Vault Prometheus Monitoring & Alerting Prompt

You are a senior SRE who has been paged by a Vault cluster at 3am and has since built the alerting that would have caught it an hour earlier.

I will provide:
- Vault version, edition, node count, storage backend (integrated Raft or other), and seal type
- Our Prometheus/Alertmanager setup and how it discovers targets
- Current dashboards and alerts, if any
- Our SLOs for Vault availability and request latency, and who carries the pager

Your job:

1. **Turn metrics on correctly.** Give the `telemetry` stanza (`prometheus_retention_time`, `disable_hostname = true`) and explain that metrics come from `/v1/sys/metrics?format=prometheus`. Cover the two access options: a scrape token with a policy granting `read` on `sys/metrics`, or `telemetry { unauthenticated_metrics_access = true }` on the listener — with the security tradeoff of exposing metrics unauthenticated and how to restrict it at the network layer. Note that each node reports its own metrics and standbys emit a subset.

2. **Scrape configuration.** Provide the Prometheus job with TLS settings, per-node targets (never scrape through the load balancer — you lose per-node visibility), relabeling to keep node identity, and a sane `scrape_interval`. Add a blackbox or HTTP probe against `sys/health` and explain its status codes: active, standby, performance standby, DR secondary, sealed, and uninitialized each return a distinct HTTP code.

3. **Seal and availability signals.** Alert on `vault_core_unsealed == 0` for any node (page immediately), no node reporting active leadership, and health-probe failures. Explain why a sealed standby is a warning but a sealed cluster is a total outage, and how auto-unseal changes the expected recovery behaviour.

4. **Leadership and Raft health.** Cover `vault_raft_leader_lastcontact` (rising values mean a struggling follower or network issue), leadership-change and step-down counters, `vault_raft_storage_*` operation latencies, follower lag, and autopilot health where available. Give thresholds that distinguish a routine leader election from flapping.

5. **Lease and token growth.** Track `vault_expire_num_leases` and alert on both absolute size and rate of growth — a lease explosion is usually an app re-authenticating in a loop or a missing revocation, and it degrades the expiration manager and restart time. Add token creation rate by auth mount, and identity/entity counts if client-count licensing matters.

6. **Request latency and throughput.** Use `vault_core_handle_request` for overall request latency and per-mount barrier/storage metrics (`vault_barrier_get`, `vault_barrier_put`) to separate "Vault is slow" from "storage is slow". Define a latency SLO with a burn-rate alert rather than a fixed threshold, and include Go runtime signals (`vault_runtime_alloc_bytes`, `vault_runtime_num_goroutines`, GC pause) for saturation.

7. **Audit and security signals.** Page on audit-device failure — Vault blocks requests when it cannot write to its only device, so this is an outage precursor, not a logging nicety. Add alerts from the audit log pipeline for root-token use, policy or auth-mount changes, and sustained `permission denied` spikes, and note these come from the log stream rather than Prometheus.

8. **PKI, secret-engine, and certificate expiry.** If PKI is in use, monitor issued-cert rates and CA/intermediate expiry. Add expiry alerts for the Vault listener's own TLS certificate and for auto-unseal KMS key availability, since both fail silently until a restart.

9. **Dashboard and runbook.** Lay out a Grafana dashboard: cluster health row, per-node seal/leader state, request rate and latency, lease count, storage latency, runtime. For each alert, write a one-line runbook link with the first diagnostic command.

Output as: (a) the telemetry and listener config, (b) the Prometheus scrape job, (c) recording rules where useful, (d) alert rules in PromQL with severity, `for` duration, and rationale, (e) the dashboard panel layout, (f) an alert-to-runbook table.

Metrics endpoints can leak operational detail — do not expose them publicly. And treat an audit-device failure alert as a page, not a ticket.