✍️ AI-Reviewed Alert Copy for Clearer Slack Notifications

I got paged at 3:14 a.m. last quarter for an alert that read Threshold exceeded: metric= qps_p99 val=812 thr=800 svc=svc-7. I stared at it for a full ninety seconds before I understood that nothing was actually on fire. The signal was real, but the copy was hostile. By the time I’d decoded it, parsed which service svc-7 mapped to, and confirmed the runbook, I’d burned the most expensive resource on-call has: a sleepy human’s attention.

That night convinced me that alert wording is a product surface, not an afterthought. And it’s exactly the kind of tedious, high-volume rewriting work where AI shines — as long as you treat it like a fast junior engineer whose drafts you review, not an oracle you trust blindly.

The problem isn’t the metric, it’s the message

Most alerting pipelines emit whatever the monitoring tool serializes by default. That’s fine for a dashboard but terrible for a Slack notification an exhausted human has to act on in seconds. A good alert message answers three questions immediately: what broke, how bad, and what do I do next. Raw payloads answer none of them.

The fix is not to hand-write hundreds of templates. It’s to let a model draft clearer versions, then have a human approve each template once before it ships. The rewrite happens at build/template time, against synthetic example payloads — never live, and never with a real workspace token or signing secret in scope.

Rewriting copy at template time, not at runtime

Here’s the pattern I use. I keep a small fixtures file of representative (fake) alert payloads, and I ask the model to produce a clearer Slack message template for each alert type. The output is a reviewed artifact that gets committed to the repo. The model never touches production data and never sees a credential.

# scripts/draft_alert_copy.py
# Runs locally/CI against synthetic fixtures. No live tokens, no real payloads.
import json
import anthropic

client = anthropic.Anthropic()  # key from env; NEVER hard-code secrets

REWRITE_PROMPT = """You are rewriting an automated Slack alert for an on-call engineer.
Rules:
- Lead with severity and the human-readable service name.
- State what changed and the current vs threshold value.
- End with one concrete next action.
- Keep it under 280 chars. No marketing tone. Use placeholders like {{service}}.
Return ONLY the message text.

Raw alert fixture:
{fixture}
"""

def draft(fixture: dict) -> str:
    msg = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=400,
        messages=[{"role": "user",
                   "content": REWRITE_PROMPT.format(fixture=json.dumps(fixture))}],
    )
    return msg.content[0].text.strip()

if __name__ == "__main__":
    fixtures = json.load(open("fixtures/alerts.json"))
    for f in fixtures:
        print(f"# {f['alert_type']}\n{draft(f)}\n")

The key discipline: the model emits placeholders ({{service}}, {{value}}), and your runtime — the part that actually has secrets — fills them in later. The AI never sees real values, and the rewrite is just text.

Pro Tip: Feed the model your severity taxonomy and a glossary mapping svc-7 to “Checkout API” in the prompt. It can’t humanize identifiers it’s never been told about, and inventing them is exactly where a junior guesses wrong.

Before and after, in Block Kit

The model’s text output is only half the job. The other half is rendering it as Block Kit so it’s scannable. Here’s a typical before — what the raw pipeline emitted:

{
  "blocks": [
    {
      "type": "section",
      "text": { "type": "mrkdwn",
        "text": "Threshold exceeded: metric=qps_p99 val=812 thr=800 svc=svc-7 env=prod ts=1718500440" }
    }
  ]
}

And the after, built from the AI-drafted, human-approved template:

{
  "blocks": [
    {
      "type": "header",
      "text": { "type": "plain_text", "text": "🔴 Checkout API — p99 latency over budget" }
    },
    {
      "type": "section",
      "fields": [
        { "type": "mrkdwn", "text": "*Current*\n812ms p99" },
        { "type": "mrkdwn", "text": "*Threshold*\n800ms" },
        { "type": "mrkdwn", "text": "*Env*\nprod" },
        { "type": "mrkdwn", "text": "*Since*\n<!date^1718500440^{time}|3:14am>" }
      ]
    },
    {
      "type": "actions",
      "elements": [
        { "type": "button", "text": { "type": "plain_text", "text": "Open runbook" },
          "url": "https://runbooks.internal/checkout-latency", "style": "primary" },
        { "type": "button", "text": { "type": "plain_text", "text": "Ack" },
          "value": "ack_checkout_p99" }
      ]
    }
  ]
}

The second one tells me everything in the time it takes to read a header. Note the <!date^...> token — Slack renders epochs in the reader’s local timezone, which matters when your on-call rotation spans continents. The model suggested it; I verified it against Slack’s formatting docs before approving, because a plausible-looking token is not a correct one.

Posting the approved template with Bolt

Once a template is reviewed and committed, your Bolt app fills the placeholders and posts. This is the only part of the flow that holds a token, and it does not call any model:

const { App } = require('@slack/bolt');

const app = new App({
  token: process.env.SLACK_BOT_TOKEN,        // real secret, server-side only
  signingSecret: process.env.SLACK_SIGNING_SECRET,
});

async function postAlert(channel, vars, template) {
  // template = the human-approved Block Kit JSON with {{placeholders}}
  const rendered = JSON.parse(
    JSON.stringify(template).replace(/\{\{(\w+)\}\}/g, (_, k) => vars[k] ?? '—')
  );
  await app.client.chat.postMessage({ channel, blocks: rendered.blocks,
    text: `${vars.service} alert` }); // text = notification fallback, required
}

Always set the top-level text field. It’s the fallback shown in notifications and on screen readers, and Slack will warn you if it’s missing.

Keep a human in the approval loop

The model drafts; a person approves. I wire this as a literal pull request: the draft_alert_copy.py output lands in a branch, and a reviewer reads every template diff before merge. This catches the two failure modes that matter — the model softening a critical alert into something that sounds routine, and the model fabricating a next-action that points at a runbook that doesn’t exist.

I’ve started routing these template diffs through an automated reviewer too, via /dashboard/code-review/, so the obvious issues get flagged before a human even looks. But the human still owns the merge. Treat AI like that fast junior: great throughput, genuinely useful drafts, zero authority to ship to a real workspace on its own.

If you want reusable starting points for the rewrite prompt itself, I keep a few in /prompts/ and bundle the polished ones in /prompt-packs/. For the actual drafting I bounce between Claude and an editor with inline AI; both are fine as long as the fixtures stay synthetic.

What this buys you

After rolling AI-reviewed copy across our top twenty alert types, median time-to-first-action on Slack pages dropped noticeably — not because the alerts fired less, but because a half-asleep human could parse them instantly. The metric thresholds didn’t change. The wording did.

The discipline is simple and worth repeating: rewrite copy at template time against fake data, render it as proper Block Kit, never let the model see a real token or payload, and require human approval before any template reaches production. For more on building out a robust Slack alerting practice, the rest of /categories/slack/ covers the surrounding tooling.

Clear alerts are cheap to produce and enormously valuable at 3 a.m. Let the fast junior write the first draft — then read it like the senior you are before it ever pages anyone.