Webhook Dedupe and Replay-Protection Receiver Design Prompt

You are a senior automation engineer who has debugged a webhook receiver that triggered the same remediation eleven times because the sender retried on a slow 200.

I will provide:
- The webhook source(s), their signing scheme, and retry behavior (timeout, max attempts, ordering guarantees)
- The delivery payload, headers, and any delivery/event IDs available
- The downstream automated action each event triggers and its blast radius
- Our storage options for dedupe state (Redis, DynamoDB, Postgres) and traffic volume

Your job:

1. **Authenticity** — specify signature verification (HMAC/timestamped), the exact header order and raw-body handling, and constant-time comparison to block forged payloads.
2. **Replay window** — define a timestamp tolerance and reject deliveries outside it, explaining the clock-skew trade-off and how stale replays are logged.
3. **Dedupe key and store** — derive a stable idempotency key from the event (delivery ID or content hash), pick a store with atomic set-if-absent, and set a TTL longer than the sender's max retry span.
4. **Ack vs. process** — separate fast acknowledgement (return 200 quickly) from the actual work via a queue, so the sender stops retrying while processing happens asynchronously and at-most-once.
5. **Out-of-order and missing events** — describe handling for reordered deliveries and a reconciliation/poll fallback for events that never arrive.
6. **Failure and DLQ** — define what happens when processing fails after ack, including a dead-letter path and a safe replay procedure that re-checks the dedupe store.
7. **Observability** — list metrics for duplicates dropped, replays rejected, signature failures, and queue depth so silent dedupe storms are visible.

Output as: a request-handling sequence diagram in prose, a dedupe-key spec, a store schema with TTL, and a metrics/alert list.

Call out every downstream action that is not naturally idempotent and require it to be gated behind the dedupe check plus a documented manual reversal for any action that did double-fire.