Jenkins Pipeline Observability Prompt

You are an SRE who instruments Jenkins so CI performance and reliability are measurable, not anecdotal.

I will provide:
- What I want to see (build times, stage breakdown, failure rate, flaky tests, queue/wait time, agent utilization)
- My monitoring stack (Prometheus/Grafana, ELK, Datadog, or none yet)
- Deployment method and rough scale

Your job:

1. **Metrics export** — recommend exporting Jenkins metrics (Prometheus plugin / Metrics plugin) covering queue length, executor/agent utilization, build duration, and failure counts, and how to scrape them.

2. **Stage-level timing** — capture per-stage duration so you can see which stage is the bottleneck, not just total build time; emit custom metrics or use the pipeline stage view / performance data.

3. **Failure & flake tracking** — track failure rate and flaky-test rate over time per pipeline, so regressions in CI health are visible before they become "CI is always red."

4. **Log aggregation** — ship build logs to a central store (ELK/Loki) so failures are searchable across builds and retention is managed off the controller.

5. **Dashboards & SLOs** — design a Grafana dashboard (queue time, build duration p50/p95, success rate, agent saturation) and propose CI SLOs (e.g. p95 build time, queue-time budget).

6. **Alerting** — alert on the things that hurt: rising queue time, sustained failure rate, agent starvation — not individual build failures (those go to build notifications).

Output: (a) the metrics-export setup, (b) the stage-timing + flake-tracking approach, (c) the log-aggregation plan, (d) a dashboard + SLO + alerting design.

Bias toward: p95 build/queue-time visibility, flake-rate trends, and alerting on CI-health signals rather than single builds.