📡 Webhook Fan-Out and Dedupe Patterns for Automation Pipelines

The webhook handler that started my appreciation for fan-out was a forty-line monster: a single GitHub push event came in, and inline, synchronously, the handler kicked a CI build, posted to Slack, updated a deployment tracker, and bumped a metrics counter. When the deployment tracker’s API got slow one afternoon, the whole handler timed out, GitHub marked the delivery failed, redelivered it — and we got duplicate CI builds and double Slack pings on top of the original slow request.

That handler was doing two hard things badly at once: fanning a single event out to multiple consumers, and not deduplicating redeliveries. Those are separate problems with separate solutions, and untangling them is what makes a webhook pipeline reliable. This is how I build them now.

Accept fast, process later

The first rule of webhook handling: your endpoint’s only job is to validate, dedupe, persist, and acknowledge — fast. Anything that does real work synchronously inside the HTTP handler is a liability, because providers enforce tight timeouts (GitHub gives you 10 seconds) and mark slow deliveries as failures, triggering redelivery.

@app.post("/webhook")
def receive(req):
    if not verify_signature(req):
        return 401
    event = parse(req.body)
    if not seen.put_if_absent(event.delivery_id):
        return 200          # duplicate redelivery, already queued
    queue.publish(event)    # hand off to async workers
    return 200              # acknowledge immediately

Acknowledge in milliseconds, do the work in workers. This single change eliminates the timeout-driven redelivery storms that cause most webhook duplication in the first place.

Dedupe at the door, not in every consumer

Notice the dedupe happens once, at ingestion, keyed on the provider’s delivery ID. Do not push deduplication down into each of the five downstream actions — that’s five chances to get it wrong. Catch the duplicate at the front door and the rest of the pipeline can assume it’s seeing each event once.

The seen store is the same atomic put_if_absent pattern that idempotency keys use: a unique constraint, a Redis SET NX with TTL, or a conditional write. TTL it to comfortably exceed the provider’s redelivery window.

Fan-out: one event, many consumers

With a single clean event on the queue, fan-out becomes a routing problem. The cleanest model is publish-subscribe: the ingestion worker publishes to a topic, and each downstream action is an independent subscriber.

def dispatch(event):
    for consumer in CONSUMERS.get(event.type, []):
        # Each consumer gets its OWN queue message and retry lifecycle
        consumer_queue.publish(consumer.name, event)

The win is isolation. The slow deployment tracker now has its own queue, its own retries, and its own failures. If it falls over, CI and Slack are completely unaffected. Compare that to the original inline handler where one slow consumer poisoned all of them.

Pro Tip: Give every fan-out consumer its own dead-letter queue, not a shared one. When the deployment tracker consumer fails, you want its poison messages isolated from the CI consumer’s. Shared DLQs turn one broken consumer into a debugging session across every event type.

Consumers must be idempotent anyway

Front-door dedupe handles provider redeliveries, but it does not make your consumers safe on its own. A consumer can crash after doing its side effect but before acknowledging its queue message, and the queue will redeliver. So each consumer still needs to be idempotent with respect to its own action.

For the Slack consumer, that might mean keying on (event.delivery_id, "slack-notify") so a redelivered queue message doesn’t double-post:

def slack_consumer(event):
    action_key = f"{event.delivery_id}:slack"
    if not actions.put_if_absent(action_key):
        return ack()
    post_to_slack(render(event))
    return ack()

Defense in depth: dedupe at ingestion and idempotency per consumer. The two layers cover different failures.

Where AI accelerates this

The boilerplate here — signature verification, the dedupe store, the consumer scaffolding — is exactly what an LLM drafts well. I use Claude or Cursor like a fast junior engineer: give it the event schema and the list of downstream actions, and it’ll generate the dispatcher and consumer stubs cleanly.

Where the human stays in charge is the routing topology and the failure policy. Which events fan out to which consumers, what happens when a consumer’s DLQ fills, whether a failed deployment-tracker update should block anything — those are decisions with operational consequences that the model can’t own. And the verification and credential handling never gets tested against real secrets by the model; generated handlers run against a local mock first. I keep vetted webhook prompts in my prompt workspace so the scaffolding starts from reviewed patterns.

Watching the pipeline

A fan-out pipeline has more moving parts than a single handler, so observability matters more. I track, per consumer: messages processed, retries, and DLQ depth. A rising DLQ depth on one consumer is the early signal that a downstream system is degrading, and I route that to the monitoring-alerts dashboard so it pages before the backlog becomes a crisis.

The metric that matters most is end-to-end: did every inbound event result in exactly one of each expected downstream action? If you can’t answer that, you can’t trust the pipeline.

Back-out and replay

Because events are persisted at ingestion and consumers are idempotent, your back-out path is replay. If a consumer was deploying a bad change, you fix the consumer and replay the affected events from the store — the idempotent consumers skip what already completed and only redo what’s needed. Persisting the raw event at the door is what makes this possible; a pipeline that processes and discards has no replay story.

Conclusion

Treat webhook ingestion and fan-out as two distinct jobs. At the door: verify, dedupe on the provider’s delivery ID, persist, acknowledge fast. Downstream: fan out to isolated, independently-retried, idempotent consumers, each with its own DLQ. Let AI draft the scaffolding, but own the routing topology, the failure policy, and the secrets.

The automation category digs into the adjacent patterns — idempotency keys, DLQ triage, and event-driven orchestration — and the prompt packs include reviewed templates for webhook handlers that don’t double-fire.