❓ Build an AI FAQ Bot in Slack That Answers From Your Engineering Docs

Every team has the question that gets asked weekly: “how do I rotate the staging database password?” The answer is documented. It is in the wiki. Nobody can find it. After answering it for the fourth time myself, I built a Slack bot that takes a question, looks it up in our docs, and answers in the channel with a link back to the source. The trick was making it honest about what it does not know, and making sure I reviewed it before unleashing it on a real workspace.

What “answers from your docs” actually means

This is not a chatbot that knows things. It is a retrieval bot: it pulls the most relevant chunks of your internal documentation, hands them to a model, and asks the model to answer using only that text. The architecture matters because it changes the failure mode. A general chatbot confidently invents a password rotation procedure. A retrieval bot that finds nothing should say “I could not find this in the docs” and tag a human. That honesty is the whole product.

Listening for questions

I trigger on app_mention so the bot only speaks when summoned. Bolt makes the event handler clean:

const { App } = require("@slack/bolt");
const app = new App({
  token: process.env.SLACK_BOT_TOKEN,
  signingSecret: process.env.SLACK_SIGNING_SECRET,
});

app.event("app_mention", async ({ event, client }) => {
  const question = event.text.replace(/<@[^>]+>/, "").trim();
  const answer = await answerFromDocs(question);
  await client.chat.postMessage({
    channel: event.channel,
    thread_ts: event.ts, // reply in-thread to keep channels tidy
    blocks: answer.blocks,
    text: answer.fallback,
  });
});

Replying in-thread with thread_ts keeps the bot from cluttering the main channel. People who care about the answer see it; everyone else is not interrupted.

Retrieval before generation

The retrieval step is where accuracy lives. I keep a simple index of doc chunks and fetch the top matches for the question. You do not need a heavyweight vector database to start; even keyword search over a well-structured docs export gets you surprisingly far.

async function answerFromDocs(question) {
  const chunks = await retrieveRelevantChunks(question, 5);
  if (!chunks.length) {
    return notFoundResponse();
  }
  const context = chunks.map((c) => `[${c.source}]\n${c.text}`).join("\n\n");
  const reply = await callModel(question, context);
  return renderAnswer(reply, chunks);
}

The [source] prefix on each chunk is deliberate. It lets the model cite where an answer came from, which is the single most important feature for trust.

Constraining the model

The prompt is strict: answer only from the provided context, cite the source, and admit uncertainty. The model is a fast junior engineer here, good at synthesizing what it is given but not to be trusted to invent procedures.

const systemPrompt = `Answer the question using ONLY the provided docs.
Cite the source name in brackets after each claim.
If the docs do not contain the answer, reply exactly:
"I could not find this in our docs."
Never invent steps, commands, or credentials.`;

That last line earns its keep. The worst outcome is a confidently wrong shell command pasted into a channel where someone copies it into production.

Pro Tip: Add a feedback reaction. When the bot answers, have it leave a placeholder and let people add a checkmark or an x. Log those reactions; they are your cheapest signal for which docs are wrong or missing.

Rendering with citations

Block Kit lets me separate the answer from its provenance so people can verify quickly:

function renderAnswer(reply, chunks) {
  return {
    fallback: reply.slice(0, 140),
    blocks: [
      { type: "section", text: { type: "mrkdwn", text: reply } },
      { type: "divider" },
      {
        type: "context",
        elements: [
          {
            type: "mrkdwn",
            text: "Sources: " + chunks.map((c) => `<${c.url}|${c.source}>`).join(" • "),
          },
        ],
      },
    ],
  };
}

I test these prompt-and-render pairs in our prompt workspace against real questions from the channel before shipping, and the stable versions go into prompts. If you would rather start from a tested set, the retrieval and citation templates in the prompt packs saved me a weekend.

The “I don’t know” path

When retrieval finds nothing, the bot does not guess. It says so and offers an escalation:

function notFoundResponse() {
  return {
    fallback: "Not found in docs",
    blocks: [
      {
        type: "section",
        text: {
          type: "mrkdwn",
          text: "I could not find this in our docs. Want me to ping <#C_DOCS_HELP>?",
        },
      },
    ],
  };
}

This is the human-in-the-loop seam. The bot handles the easy 80 percent and hands the hard 20 percent to a person, rather than fabricating an answer.

Security: verify everything inbound

Because this bot listens on a public Events API endpoint, every request must be verified. Bolt checks the signing secret for you, validating the X-Slack-Signature HMAC against the raw body and the timestamp. If you ever build the endpoint by hand, do that verification yourself and reject stale timestamps to block replay attacks. And the model never sees a token, a session cookie, or anything secret. It sees questions and doc text, nothing more. Before this bot went into a real workspace, a teammate and I read through twenty sample answers together to make sure it failed safely.

Where it fits

A docs FAQ bot pairs naturally with the rest of an ops-bot fleet. It is the front door; the deeper automations like incident response handle what happens when the answer is “page someone.” For building the bot itself, AI assistants speed up the plumbing. I used ChatGPT for the retrieval scaffolding and Claude for tuning the answer prompt, then reviewed every line. The broader patterns live in the Slack category.

Conclusion

The value of a docs FAQ bot is not that it is clever. It is that it is honest: it answers when it can cite a source and stays quiet when it cannot. Build retrieval first, constrain the model to the context, render citations, verify every inbound signature, and keep secrets out of the prompt. Have a human read its early answers before it speaks to your whole workspace. Done that way, it turns “where is that documented?” into a one-line question with a linked answer.