🚨 Automation Error: Webhook 504 Timeout Triggers a Retry Storm

Overview

A webhook receiver that used to work starts returning 504 Gateway Timeout, and the sender — doing exactly what it is designed to do — retries. Each retry adds load to an already-slow consumer, which makes it slower, which produces more timeouts and more retries. The result is a self-amplifying retry storm visible in the receiver’s proxy logs and the sender’s delivery dashboard:

# Receiver side (reverse proxy in front of the webhook handler)
upstream timed out (110: Connection timed out) while reading response header from upstream,
  request: "POST /webhooks/github HTTP/1.1", upstream: "http://10.0.3.11:8080/webhooks/github"
2026-07-08T14:22:07Z  504  POST /webhooks/github  30001ms

# Sender side (delivery log) — same event id redelivered
delivery 4f1c... attempt 1: 504 (30.0s)  -> will retry
delivery 4f1c... attempt 2: 504 (30.0s)  -> will retry
delivery 4f1c... attempt 3: 200 (0.9s)   # finally acked — but attempts 1 & 2 also ran to completion server-side

The subtle damage: the handler often finished the work on attempts 1 and 2 but took longer than the proxy or sender timeout to respond, so the sender never saw the ack and redelivered. Now the same event is processed two or three times.

Symptoms

• Webhook endpoint returns 504 (or 502) under load while the app itself logs successful processing.
• Sender delivery dashboard shows escalating retries and duplicate event_ids for the same event.
• Latency climbs monotonically until the receiver is saturated; recovery only happens when the sender gives up or you shed load.
• Downstream side effects are duplicated (double notifications, double writes, double CI triggers).
• The handler’s own timing shows work completing just after the proxy/sender timeout threshold (e.g. 30s).
• Request queue depth or worker pool utilization at the receiver is pinned at 100%.

Common Root Causes

• Synchronous processing inside the request. The handler does the full job (DB writes, API calls, image builds) before responding, so response time is coupled to work time and exceeds the sender’s timeout.
• Sender timeout shorter than worst-case processing. GitHub, Stripe, and most providers cut off after ~10-30s and then retry; any handler that can exceed that will be redelivered.
• No fast ack / no queue. Without a “receive, enqueue, 200 immediately” pattern, a burst of events runs concurrently in the request path and starves the worker pool.
• Non-idempotent handler. Because the work is not keyed on event_id/delivery id, each redelivery re-executes side effects instead of deduping.
• Downstream dependency slowdown. A slow database or third-party API inflates handler time past the timeout, converting a latency blip into a retry storm.
• Retries with no backoff or cap on your side. If you also re-enqueue failed work aggressively, your internal retries compound the sender’s retries.

Diagnostic Workflow

Confirm where the time is spent — proxy vs app — by comparing proxy access time to the handler’s own timing:

# Reverse proxy: how long did the upstream take, and what status was returned?
grep 'POST /webhooks/' /var/log/nginx/access.log | awk '{print $NF, $status}' | sort | uniq -c | tail
kubectl logs deploy/webhook-receiver --tail=200 | grep -E 'event_id|duration_ms'

Measure the handler’s real end-to-end latency directly, bypassing the sender:

curl -sS -o /dev/null -w 'status=%{http_code} total=%{time_total}s\n' \
  -X POST https://hooks.example.com/webhooks/github \
  -H 'Content-Type: application/json' --data @sample-event.json

Look for duplicate deliveries of the same event id in your logs (the signature of a storm):

kubectl logs deploy/webhook-receiver --since=15m | grep -oE 'event_id=[a-f0-9]+' \
  | sort | uniq -c | sort -rn | head    # counts > 1 = redelivered events

Check the proxy/upstream timeout and the worker saturation:

grep -iE 'proxy_read_timeout|proxy_send_timeout' /etc/nginx/nginx.conf /etc/nginx/conf.d/*
kubectl logs deploy/webhook-receiver --tail=50 | grep -iE 'queue|pool|reject|429'

If a queue exists, inspect its depth to distinguish “slow handler” from “no capacity”:

redis-cli LLEN webhook:incoming        # or your broker's depth metric

Example Root Cause Analysis

A GitHub webhook receiver ran CI-trigger logic synchronously: on each push it cloned metadata, wrote to Postgres, and called an external build API before returning 200. Normal events took ~1.5s. During a busy afternoon Postgres latency rose, pushing handler time to ~34s — just past GitHub’s delivery timeout. GitHub marked deliveries as failed at 30s and redelivered them; each redelivery ran the entire handler again, adding load to the same slow Postgres and worker pool.

Proxy logs showed a wall of 504 ... 30001ms. The duplicate-id check (uniq -c on event_id) showed many events processed two or three times, and the CI system logged duplicate builds for single commits. Root cause: the handler coupled its HTTP response to long, non-idempotent work, and the work time crossed the sender’s fixed timeout — so every slow event became two or three events, amplifying the very slowness that caused it.

The fix was to invert the handler: validate the signature, write the raw event to a queue keyed by GitHub’s X-GitHub-Delivery id, and return 200 in under 200ms. A separate worker pool drained the queue and deduped on the delivery id, so redeliveries became no-ops. Handler p99 dropped below the timeout, retries stopped, and the storm could not re-form because acks no longer waited on the work.

Prevention Best Practices

• Ack fast, process async. Validate, enqueue, and return 200 in well under the sender’s timeout (aim for < 1s). Never do the real work in the request path.
• Make handlers idempotent on the delivery id. Dedupe on the provider’s delivery/event id (X-GitHub-Delivery, Stripe event.id) so any redelivery is a safe no-op.
• Know and beat the sender’s timeout. Look up each provider’s delivery timeout and keep worst-case response time comfortably under it, not near it.
• Bound your own retries. Use capped exponential backoff with jitter and a dead-letter queue for internal reprocessing so you never compound the sender’s retries.
• Add backpressure, not infinite concurrency. Cap in-flight work and shed load with a fast 429/503 (which well-behaved senders back off on) rather than letting requests pile up into timeouts.
• Alert on duplicate-delivery rate and handler p99. A rising redelivery count or a p99 approaching the sender timeout is the early warning before a full storm.

Quick Command Reference

# Proxy status + upstream time distribution
grep 'POST /webhooks/' /var/log/nginx/access.log | awk '{print $status, $NF}' | sort | uniq -c

# Real handler latency, bypassing the sender
curl -sS -o /dev/null -w 'status=%{http_code} total=%{time_total}s\n' \
  -X POST https://hooks.example.com/webhooks/github --data @sample-event.json

# Duplicate deliveries = storm signature
kubectl logs deploy/webhook-receiver --since=15m \
  | grep -oE 'event_id=[a-f0-9]+' | sort | uniq -c | sort -rn | head

# Proxy upstream timeout setting
grep -iE 'proxy_read_timeout|proxy_send_timeout' /etc/nginx/nginx.conf /etc/nginx/conf.d/*

# Queue depth: slow handler vs no capacity
redis-cli LLEN webhook:incoming

Conclusion

A webhook 504 retry storm is a feedback loop: a handler that does slow work in the request path crosses the sender’s fixed timeout, the sender retries, and each retry re-runs the work and deepens the slowness. Break the loop by decoupling the ack from the work — enqueue and return 200 in milliseconds — and by making processing idempotent on the provider’s delivery id so redeliveries are harmless no-ops. Add backpressure and duplicate-rate alerting, and the storm cannot form: acks stay fast even when the work behind them is slow.