Slack Knowledge Q&A Bot Design Prompt

You are a senior platform engineer who has built a RAG-based Q&A bot serving internal SRE / platform questions with measurable knowledge-base improvement over time.

I will provide:
- Knowledge sources (Confluence / Notion / GitHub markdown / Google Docs / runbooks repo / postmortems)
- LLM choice (Claude / GPT / Gemma local)
- Embedding model preference
- Volume estimate (questions/day)
- Confidentiality concerns

Your job:

1. **Architecture**:
   - **Ingestion** — periodic crawl of each source; chunk by section; generate embeddings; upsert to vector DB
   - **Query** — Slack message → bot → embed query → retrieve top-K chunks → LLM with chunks as context → response with sources
   - **Feedback** — thumbs-up/down reactions; track confidence; refresh stale answers

2. **Source prioritization**:
   - **Tier 1** (authoritative): official runbooks, architecture docs, OpenAPI specs
   - **Tier 2** (contextual): recent postmortems, design docs, ADRs
   - **Tier 3** (supplemental): Slack message search results, GitHub issue resolutions
   - Weight retrieval results by tier; downweight Tier 3 if Tier 1 has the answer

3. **Chunk design**:
   - **By section** (markdown headers), not arbitrary size
   - Include parent-section context in chunk metadata
   - Source URL + author + last-modified date as metadata
   - For runbooks: include the runbook ID + the step number

4. **Query handling**:
   - Slack message → detect if it's a question (vs chitchat)
   - Trigger: `@bot <question>` OR opt-in channel with auto-detect
   - For ambiguous queries, ask a clarifying question (don't answer wrong with confidence)

5. **Response format** — Block Kit:
   - **Answer** — 1-2 paragraphs, plain language
   - **Sources** — bullet list with source URL + freshness ("Last updated 2 weeks ago")
   - **Confidence** — low / medium / high; if low, suggest "you might want to verify with @team"
   - **Feedback** — 👍 / 👎 reactions wired to your store

6. **Freshness signals**:
   - Source last-modified > 6 months → "this doc may be outdated" footer
   - Conflicting sources → "I found conflicting info; here are both" with two answers
   - Question about a service deprecated in code → "this service was removed; see #migrations"

7. **Privacy & confidentiality**:
   - Strip secrets / API keys from sources before embedding (regex pre-filter)
   - Don't index private DMs
   - Respect source-level ACLs (don't surface a Confluence space the asker can't read)
   - For air-gapped: use local LLM (Gemma) and self-hosted vector DB

8. **Feedback loop**:
   - 👍 → boost source ranking for similar questions
   - 👎 → log for review; if frequent on same query, refresh source ranking
   - Bot DMs sources owners when their docs get 👎 frequently
   - Weekly admin report: top 10 unanswered / poorly-answered questions → docs gap analysis

9. **Anti-patterns to avoid**:
   - Confident-sounding answers with no source
   - Stale sources surfaced without freshness warning
   - Indexing sensitive data into the vector DB
   - Ignoring 👎 feedback
   - Replacing humans (bot answers, no escalation path)

10. **Escalation** — when confidence is low, when no source exists, when topic is sensitive:
   - Bot says "I don't have a high-confidence answer; @team can help"
   - Routes to the channel for human follow-up
   - Logs the gap for documentation owners

11. **Metrics**:
   - Questions answered / day
   - User satisfaction (% positive reactions)
   - Knowledge gap rate (% questions with no good source)
   - Source freshness distribution
   - Topics with lowest confidence (docs gaps)

Output as: (a) ingestion pipeline, (b) source tier prioritization, (c) chunk + metadata schema, (d) query handling flow, (e) Block Kit response format, (f) freshness signal rules, (g) privacy filters, (h) feedback loop design, (i) metrics dashboard.

Bias toward: cite sources every time, surface freshness, escalate when uncertain, treat the bot as a docs-discoverability tool not a replacement for docs.