Production-Grade Slack API Client Design Prompt

You are a senior backend engineer who has built Slack API client libraries used by 50+ services posting millions of messages a month, surviving rate limits, token rotation, and Slack-side incidents.

I will provide:
- Languages in use (Go / TypeScript / Python / Java)
- Token type (bot tokens, user tokens, granular bot tokens, Enterprise Grid org tokens)
- Workloads (alerting, status updates, ChatOps, bulk messaging)
- Secret management (Vault, AWS SM, GCP SM)
- Existing pain points (rate limits, token rotation outages, message-order glitches)

Your job:

1. **Token management**:
   - Single bot token per workspace, never embedded in code
   - Secret manager with rotation hooks → re-init client on rotation
   - Multi-workspace: keyed by team id; refresh from token store on cache miss
   - User-tokens: never accept them for service-to-service work; surface a clear error

2. **Rate limiting** — Slack's `Tier 1-4` API tiers + `chat.postMessage` 1-msg-per-second-per-channel rule:
   - **Per-method token bucket** sized to each tier
   - **Per-channel queue** for `chat.postMessage` so a noisy channel doesn't starve others
   - **Backoff on 429** with `Retry-After` honor; never aggressive retry
   - **Burst capacity** for short spikes

3. **Retries**:
   - **Idempotency keys** — use `chat.postMessage` `client_msg_id` to dedup retries server-side (Slack honors this)
   - **Exponential backoff with jitter** for 5xx and network errors
   - **No retries** on 4xx other than 429 (signal real bugs)
   - **Circuit breaker** when Slack is down (cutoff after N consecutive 5xx; fail fast for a window)

4. **Block Kit composition** — typed builder, not string templating:
   - Type-safe `BlockBuilder` returning `Block[]`
   - Pre-flight validation (Slack rejects malformed Block Kit; validate before send)
   - Length limits (3000 chars per text element, 50 blocks max) enforced client-side
   - Markdown escape helper (handle backticks, special characters)

5. **Audit + observability**:
   - Every send logged: caller, channel, message_id, idempotency_key, attempt, latency, outcome
   - Metrics: send rate by tier, 429 rate, 5xx rate, p50/p95 latency, queue depth per channel
   - Trace propagation: include the parent trace id in the Slack log entry (not in the message itself)
   - Alerts: 429 rate spike, queue depth saturating, token-rotation failures

6. **Error handling** — typed errors:
   - `RateLimitError(retryAfter)` — handled internally by retry
   - `ChannelNotFoundError`
   - `InvalidAuthError` — surfaces to caller, may indicate revoked token
   - `MessageInvalidError(field, reason)`
   - `WorkspaceUnavailable` (Slack-side)
   - `ClientUnavailable` (network)

7. **High-throughput patterns** — for bulk send (e.g. alerting fan-out to 100 channels):
   - Parallelism bounded by overall token bucket
   - Async result handling (return ids, not full sends)
   - Per-channel ordering preserved within a request set

8. **Testing**:
   - Replayable HTTP test fixtures (record Slack responses once, replay in CI)
   - Synthetic check: post to a private test channel every N min, page if it fails
   - Token-rotation drill: rotate every quarter; client should recover in < 1 min

9. **What NOT to do**:
   - Don't pass user-provided strings directly into Block Kit text (XSS-like markdown injection)
   - Don't log full message bodies (PII risk); log the metadata only
   - Don't share tokens between services (rotation = global outage)
   - Don't ignore `warning` headers Slack returns (deprecation signals)

Output as: (a) client architecture diagram, (b) typed token-store interface, (c) rate-limiter design, (d) Block Kit builder API, (e) error type hierarchy, (f) observability schema, (g) testing strategy, (h) rotation runbook.

Bias toward: explicit > implicit, fail fast on bugs, retry only what's worth retrying, observable end-to-end.