📨 Route Customer Feedback to the Right Slack Channel With AI Triage

Customer feedback arrived everywhere: support tickets, a feedback form, app-store reviews, the occasional angry tweet. It all landed in one firehose channel that everyone muted within a week. The bug reports got lost next to the feature requests next to the “great app!” notes. I wanted each piece of feedback to land in the channel that could actually act on it. So I built a triage bot: a model reads each item, classifies it, and routes it. The careful part was making sure low-confidence calls went to a human, not silently to the wrong place.

The routing problem

Routing is classification plus a destination map. Bug reports go to #bugs, feature requests to #product, billing complaints to #support-billing, and praise to #wins so the team sees it. A human can sort these instantly, but a human cannot sort hundreds a day. A model can, and classification into a fixed set of buckets is one of the most reliable things a model does, because the answer space is small and well-defined.

Ingesting feedback

Feedback arrives via a webhook from our forms and integrations. The endpoint validates the source, then enqueues the item for triage:

const { App, ExpressReceiver } = require("@slack/bolt");
const receiver = new ExpressReceiver({ signingSecret: process.env.SLACK_SIGNING_SECRET });
const app = new App({ token: process.env.SLACK_BOT_TOKEN, receiver });

// Our own feedback intake (not from Slack) gets its own HMAC check:
receiver.router.post("/feedback", verifyIntakeSignature, async (req, res) => {
  await triageAndRoute(req.body);
  res.sendStatus(200);
});

The verifyIntakeSignature middleware checks an HMAC on our intake webhook. Never trust an inbound payload just because it hit your endpoint; verify the signature on every source, Slack or otherwise.

Classifying with the model

The classification prompt is strict: choose one bucket from a fixed list, and return a confidence. The model is a fast junior triager, fast and consistent but occasionally wrong on the ambiguous ones.

const prompt = `Classify this customer feedback into exactly one category:
bug, feature_request, billing, praise, other.
Return JSON: { "category": "...", "confidence": 0-1, "reason": "..." }.

Feedback: "${feedback.text}"`;

Returning a confidence score is what makes the human-in-the-loop possible. High confidence routes automatically; low confidence goes to a person.

async function triageAndRoute(feedback) {
  const result = await classify(feedback);
  if (result.confidence >= 0.8) {
    await routeTo(CHANNEL_MAP[result.category], feedback, result);
  } else {
    await routeTo("#triage-review", feedback, result, { needsHuman: true });
  }
}

Pro Tip: Tune the confidence threshold by watching the review channel. If #triage-review is empty, your threshold is too low and the bot is guessing on ambiguous items. If it is flooded, the threshold is too high. Aim for a handful a day landing in front of a human.

Rendering routed feedback

Each routed item is a compact card with the original text, the category, and a way to correct it:

function buildFeedbackBlocks(feedback, result, opts = {}) {
  const blocks = [
    {
      type: "section",
      text: { type: "mrkdwn", text: `*New feedback* (${result.category}, ${(result.confidence * 100).toFixed(0)}%)\n>${feedback.text}` },
    },
  ];
  if (opts.needsHuman) {
    blocks.push({
      type: "actions",
      elements: ["bug", "feature_request", "billing", "praise"].map((c) => ({
        type: "button",
        text: { type: "plain_text", text: c },
        action_id: `route_${c}`,
        value: JSON.stringify(feedback),
      })),
    });
  }
  return blocks;
}

The human correction loop

When a low-confidence item shows up in #triage-review, a human clicks the right category and the bot reposts it to the correct channel. Those corrections are gold: I log every one as a labeled example to evaluate and improve the prompt over time.

app.action(/route_.+/, async ({ ack, body, action, client }) => {
  await ack();
  const category = action.action_id.replace("route_", "");
  const feedback = JSON.parse(action.value);
  await client.chat.postMessage({
    channel: CHANNEL_MAP[category],
    blocks: buildFeedbackBlocks(feedback, { category, confidence: 1, reason: "human-routed" }),
    text: "Routed feedback",
  });
  await logCorrection(feedback, category); // training signal for prompt tuning
});

This is the seam where AI and human divide the work cleanly. The model handles the obvious majority; people handle the ambiguous minority and, in doing so, make the bot better. I never let the model route something it was unsure about straight into a customer-facing or decision-driving channel.

Tokens and signatures

The bot token lives in an environment variable and never touches the model; the model sees only the feedback text. Every inbound request, both Slack interactions and our own intake webhook, has its signature verified. Bolt checks Slack’s signing secret automatically against the raw body and timestamp; our intake middleware does the same for our own source. Before this went live in our real workspace, I ran a week of feedback through it in shadow mode and reviewed the routing against what a human would have done.

I tuned the classification prompt in the prompt workspace using real historical feedback, then saved the working version to prompts. The classification templates in the prompt packs are a good starting point. For building the handlers I used ChatGPT and Claude, reviewing each change, and the broader patterns live in the Slack category.

Conclusion

A feedback triage bot turns a muted firehose into signal that lands where someone can act on it. Classify into a fixed set of buckets, return a confidence, auto-route the confident majority, and send the ambiguous rest to a human who corrects it and improves the model in the process. Verify every inbound signature, keep tokens out of the prompt, and shadow-test before going live. The model sorts; people handle the edges. That balance is what keeps the routing accurate enough to trust.