Alertmanager Inhibition & Silence Strategy Prompt
Design inhibition rules and silences that suppress downstream noise — when a node dies, don't also page for every pod on it — without ever muting the alert that actually matters.
- Target user
- On-call leads tuning Alertmanager to cut cascade noise
- Difficulty
- Advanced
- Tools
- Claude, ChatGPT
The prompt
You are an alerting architect who has untangled pager storms where one root cause fired 200 alerts. You wield inhibition like a scalpel: suppress the symptoms, never the cause. I will provide: - The alert taxonomy (names, severities, labels) - Real incident examples where one failure cascaded into many pages - Current Alertmanager `inhibit_rules` (if any) and routing tree - The label set shared between cause and effect alerts (e.g. instance, cluster, node) Your job: 1. **Map cause → effect** — for each cascade, identify the source alert (NodeDown, ClusterUnreachable, DatabaseDown) and the dependent alerts it should mute (PodNotReady, TargetDown, HighLatency on that node). 2. **Write inhibit_rules** — `source_matchers`, `target_matchers`, and the critical `equal:` labels that scope suppression to the SAME entity. Explain how a missing/incorrect `equal` either over-suppresses globally or does nothing. 3. **Severity inhibition** — suppress `warning` for a service when its `critical` is already firing; show the rule and the ordering implications. 4. **Maintenance silences** — `amtool silence add` patterns for deploys/maintenance windows, with TTLs, comments, and a creator id; and how to script silences from CI for planned changes. 5. **Guard against over-suppression** — what must NEVER be inhibited (the Watchdog, paging SEV1s), and how to detect a silence that's hiding a real outage (a meta-alert on long-lived/over-broad silences). 6. **Validation** — replay a past incident through `amtool` to confirm the inhibition produces exactly one actionable page. Output: (a) the `inhibit_rules` YAML with comments per rule, (b) a cause→effect suppression matrix, (c) amtool silence scripts for maintenance, (d) a meta-alert for dangerous silences, (e) a replay/test procedure. Bias toward: scoping every rule with `equal`, suppressing symptoms only, and treating a too-broad silence as an incident in itself.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Alertmanager Silence Automation via amtool & API Prompt
Automate creating, expiring, and auditing Alertmanager silences around deploys and maintenance windows using amtool and the v2 API, with matchers that don't over-silence.
-
Alertmanager PagerDuty Receiver Integration Prompt
Wire Prometheus Alertmanager to PagerDuty with correct severity mapping, dedup keys, custom details, and auto-resolve so on-call pages are actionable and noise-free.
-
Alertmanager HA Cluster & Gossip Mesh Design Prompt
Design and debug a highly available Alertmanager cluster — gossip mesh, notification deduplication across replicas, and split-brain avoidance — so alerts fire exactly once during failures.
-
Alertmanager Webhook Receiver Integration Prompt
Build a robust custom webhook receiver for Alertmanager — parsing the v4 payload, handling firing/resolved, verifying signatures, and bridging alerts into ticketing, automation, or chatops safely.
More Prometheus & Monitoring prompts & error guides
Browse every Prometheus & Monitoring prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.