Slack Scheduled Job (Cron) Failure Notification & Retry Prompt
Design Slack notifications for failed scheduled jobs and CronJobs that report failure context, missed runs, and offer guarded retry/skip actions.
- Target user
- Engineers operating batch jobs, cron, and Kubernetes CronJobs
- Difficulty
- Beginner
- Tools
- Claude, ChatGPT
The prompt
You are a senior platform engineer who has wrangled hundreds of scheduled jobs and built Slack notifications that make batch failures obvious and quick to act on. I will provide: - Where our jobs run (Kubernetes CronJobs, system cron, Airflow, cloud schedulers) - How job results surface today (exit codes, logs, a wrapper script) - Which jobs are critical vs best-effort, and their owners - Slack constraints (webhook or bot token, channel layout) - Pain points (silent failures, alert spam from flaky jobs, no missed-run detection) Your job: 1. **What deserves an alert** — failed runs of critical jobs always; best-effort jobs only after N consecutive failures; and crucially, missed runs (a job that never started when scheduled) for critical jobs. 2. **Missed-run detection** — explain a heartbeat/dead-man's-switch approach: each successful run pings a tracker, and a watcher alerts when an expected run is overdue. 3. **Message design** — Block Kit: header (job name + schedule + status emoji), section with exit code, duration, run start time, and owner mention; context block with a link to logs and the run history. 4. **Failure context** — include the last lines of stderr, the exit code meaning if known, and whether prior runs succeeded (e.g. "first failure in 30 days" vs "5th in a row"). 5. **Action buttons** — Retry Now (guarded: confirms it is safe to re-run, respects idempotency), Skip This Run, View Logs, and Acknowledge. 6. **Flap suppression** — for jobs that fail intermittently, batch repeated failures into a single updating message rather than one per run. 7. **Recovery notices** — post a quiet "recovered after N failures" message when a previously failing job succeeds, then resolve the thread. 8. **Validation** — confirm that a genuinely missed run (job never scheduled) produces an alert, since plain failure-only monitoring would miss it. Output as: (a) the job-wrapper that reports success/failure to Slack, (b) the heartbeat/missed-run watcher logic, (c) Block Kit JSON for one failure message, (d) the critical-vs-best-effort job config schema, (e) a rollout plan for one job first. Bias toward: catching silent and missed runs, quiet for flaky best-effort jobs, retries always guarded by idempotency.
Run this prompt with AI
Test it, get an AI-improved version, or compare models — live in the Prompt Workspace. No copy-paste.
Related prompts
-
Prometheus Alert Rule to Slack Template Prompt
Author Prometheus alerting rules whose labels and annotations render directly into a high-signal Slack message — required summary/runbook/dashboard fields, severity labels, and a Go template that produces a scannable Block Kit-ready payload.
-
Slack Threading and Broadcast Strategy Prompt
Design a consistent threading model for an ops/incident channel — when a bot replies in-thread, when it broadcasts to channel, and how alert updates, acks, and resolves stay grouped without flooding the main feed.
-
Slack Alert Fatigue Tuning Audit Prompt
Audit a noisy Slack alert channel, identify which alerts are ignored, duplicated, or non-actionable, and produce a concrete tuning plan that cuts volume without dropping anything that actually matters.
-
Slack Grafana Alerting Webhook Routing Prompt
Build a Grafana unified-alerting contact point that posts clean, severity-aware Slack messages with deep-linked panels, folder-based channel routing, and resolved-state updates.
More Slack prompts & error guides
Browse every Slack prompt and troubleshooting guide in one place.
Reading prompts? Get all 500 in one free PDF
500 battle-tested, copy-paste AI prompts engineered by a senior systems engineer — every one with fill-in placeholders and safety/back-out notes. Drop your email and it's yours.
- 500 prompts: Linux · Kubernetes · Terraform · OpenStack · GitLab · Docker · Monitoring · Incident Response
- Instant PDF download — yours free, forever
- Plus one practical AI-workflow email a week (no spam)
Single opt-in · unsubscribe anytime · no spam.