kube-state-metrics & cAdvisor Alerting Prompt
Build the essential Kubernetes workload alerts from kube-state-metrics and cAdvisor — CrashLoopBackOff, OOMKills, pending pods, throttling, and PVC pressure — with correct joins and no double-paging.
- Target user
- Platform engineers writing Kubernetes alerting rules on a Prometheus stack
- Difficulty
- Intermediate
- Tools
- Claude, ChatGPT
The prompt
You are a Kubernetes SRE who has written the alerting rules teams actually keep — the ones that catch real workload failures without paging on every rollout.
I will provide:
- My cluster setup (kube-state-metrics + cAdvisor via kubelet, Prometheus or Operator)
- Workload types (Deployments, StatefulSets, Jobs, DaemonSets) and their criticality
- Current alert pain points (noise during deploys, missed OOMs)
Your job:
1. **Source-of-truth map** — clarify which signal comes from kube-state-metrics (`kube_pod_status_phase`, `kube_pod_container_status_restarts_total`, `kube_deployment_status_replicas_*`, `kube_pod_container_status_waiting_reason`) vs. cAdvisor (`container_cpu_cfs_throttled_periods_total`, `container_memory_working_set_bytes`, `container_oom_events_total`). Note overlaps and which to trust.
2. **Write the core alerts**:
- CrashLoopBackOff via `kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}`
- Restart storms via `rate(kube_pod_container_status_restarts_total[15m])`
- OOMKills via `container_oom_events_total` / `reason="OOMKilled"`
- Pods Pending/Unschedulable too long
- CPU throttling ratio from cAdvisor
- Memory working set near limit (using `kube_pod_container_resource_limits` joins)
- Deployment replica mismatch (available < desired) with a `for:` that tolerates normal rollouts
3. **Joins done right** — show the `* on(...) group_left(...)` patterns to attach limits, owner workload, and namespace labels, and warn about the label-name mismatches (`pod` vs `pod_name`, version drift) that silently break joins.
4. **Deploy-noise suppression** — use `for:` durations and `kube_deployment_status_observed_generation` so rollouts don't page; suppress replica-mismatch during an in-progress rollout.
5. **Severity tiering** — what pages vs. what's a ticket; how to inhibit pod-level alerts when the whole node/namespace is down.
6. **Multi-tenancy** — scope or template rules per namespace/team without copy-paste sprawl.
Output as: (a) the alerting rules YAML grouped by signal, (b) a join cheat-sheet for label matching, (c) per-alert severity + rationale, (d) inhibition rules to prevent cascade paging.
Bias toward: actionable pages only, rollout-aware suppression, joins that survive label drift.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus Meta-Monitoring & Self-SLO Design Prompt
Build the monitoring-of-the-monitoring layer: alerts and SLOs that tell you when Prometheus itself is unhealthy — scrapes lagging, rules failing, WAL growing, or the whole instance dead — so your blind spots do not become silent outages.
-
Prometheus Rule Unit Testing with promtool Prompt
Generate promtool unit test files (test_rules.yml) that assert alert firing, recording-rule output, and label propagation for Prometheus rule groups in CI.
-
PromQL absent_over_time Gap Detection Prompt
Design absent()/absent_over_time() expressions that detect missing metrics, scrape gaps, and label-scoped absence without false-firing during restarts or expected idle periods.
-
Prometheus Staleness & Stale Markers Prompt
Understand and debug Prometheus staleness handling — stale markers, the 5-minute lookback, disappearing targets, and how staleness interacts with alert rules and absent().
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.