🚨 Automation Error Guide: 'Idempotency key conflict' Duplicate Event Processed Twice

Overview

An idempotency conflict means the same logical operation arrived more than once and your system either rejected the duplicate or, worse, processed it twice. Most automation transports (queues, webhooks, retried HTTP calls) are at-least-once: a redelivery can repeat any event. Idempotency keys exist to make a repeated request a no-op. A “conflict” surfaces either as a deliberate 409 Conflict when a key is replayed with a different body, or as a data anomaly (a charge applied twice, a row inserted twice) when no dedup guard exists at all.

You will see the explicit conflict in an API/job log:

WARN idempotency conflict key=evt-9f3a-2026 status=409 reason="key reused with different request body"
INFO http "POST /charges HTTP/1.1" 409 58

Or the silent double-processing in your data/audit log:

ERROR duplicate side effect: order=ORD-7781 charged twice (txn=txn_aaa, txn=txn_bbb) within 800ms

It occurs whenever an upstream retries or a broker redelivers — on webhook retries, queue redelivery after a missed ack, or client-side request retries on a timeout. A pipeline that’s correct under single delivery breaks the instant a retry repeats an event.

Symptoms

• Explicit 409 Conflict / “idempotency key conflict” responses in logs.
• Duplicate rows, double charges, or doubled counters traced to one source event.
• Two records with the same business key but different system IDs created milliseconds apart.
• Retries from a webhook provider or queue line up with the duplicate effects.

# Find duplicate business keys created close together
grep -RniE "duplicate|idempotency|409 Conflict" /var/log/orders-svc/*.log | tail

WARN idempotency conflict key=evt-9f3a status=409
ERROR duplicate side effect order=ORD-7781 charged twice

# Are two records sharing one event id?
psql -c "SELECT event_id, count(*) FROM processed_events GROUP BY 1 HAVING count(*)>1 LIMIT 5;"

 event_id   | count
------------+-------
 evt-9f3a   |     2

Common Root Causes

1. No dedup store / key never checked

The handler simply does the work every time it’s called; there is no record of which keys were already processed.

grep -RniE "idempotency|dedup|already_processed|seen_key|INSERT .* processed_events" ./src | head

(no matches)

No idempotency logic at all means every redelivery re-runs the side effect.

2. Idempotency key is missing or not propagated

The upstream sends a key but the consumer reads the wrong header/field (or the key is lost across a hop), so each delivery looks unique.

grep -RniE "Idempotency-Key|idempotencyKey|x-event-id|messageId" ./src | head

src/handler.ts:12: const key = req.headers['idempotency-key']  // provider sends X-Event-Id

Reading a header the sender never sets yields undefined, defeating dedup.

3. Race condition: two deliveries processed concurrently

Two copies of the same event are received simultaneously; both check “not seen” before either writes the marker, so both proceed.

# Two near-simultaneous handler starts for one key
grep -RniE "evt-9f3a" /var/log/orders-svc/*.log | grep -i "begin process" | head

13:02:11.418 begin process key=evt-9f3a worker=w3
13:02:11.421 begin process key=evt-9f3a worker=w7

3ms apart on two workers = a check-then-act race; the dedup write must be atomic.

4. Key chosen from a non-unique field

The idempotency key is derived from something that isn’t actually unique per operation (e.g., user id + day), so distinct operations collide → 409, or repeats share a key incorrectly.

grep -RniE "idempotencyKey =|key = .*\+|hash\(" ./src | head

src/charge.ts:9: const key = `${userId}-${today}`   // two charges same day collide

A coarse key both falsely conflicts distinct operations and falsely merges repeats.

5. Dedup record TTL expired before a late redelivery

The dedup entry is evicted (short TTL) before a delayed retry arrives, so the late duplicate is treated as new.

redis-cli TTL idem:evt-9f3a

(integer) -2

TTL -2 (key gone) means the dedup marker expired; a retry after that window re-processes.

6. Non-atomic check-and-set against the store

The handler does GET then SET as two operations instead of an atomic SET NX / unique-constraint insert, leaving a window for duplicates.

grep -RniE "redis.*get|redis.*set|SET NX|INSERT .* ON CONFLICT|unique" ./src | head

src/dedup.ts:7: const seen = await redis.get(key)
src/dedup.ts:9: if (!seen) await redis.set(key, '1')   // GET then SET = race window

Diagnostic Workflow

Step 1: Confirm it’s a duplicate of one logical event

psql -c "SELECT event_id, count(*) FROM processed_events GROUP BY 1 HAVING count(*)>1 LIMIT 10;"

A count > 1 per event_id (or duplicate business keys) confirms double-processing.

Step 2: Determine the source of the repeat

grep -RniE "evt-9f3a|ORD-7781" /var/log/*/*.log | grep -iE "redeliver|retry|delivery_count|attempt" | tail

Identify whether the repeat came from a webhook retry, queue redelivery, or client retry — that’s the at-least-once source you must guard against.

Step 3: Verify a key is present and read correctly

grep -RniE "Idempotency-Key|x-event-id|messageId|idempotencyKey" ./src

Confirm the consumer reads the exact field the sender sets, and that the key uniquely identifies the operation.

Step 4: Inspect the dedup store and TTL

redis-cli TTL idem:evt-9f3a
redis-cli EXISTS idem:evt-9f3a

A missing/expired key during a late retry points at TTL; ensure the TTL exceeds the maximum redelivery window.

Step 5: Check that the check-and-set is atomic

grep -RniE "SET .* NX|ON CONFLICT|unique constraint|getset|INSERT .* RETURNING" ./src

Replace GET then SET with an atomic SET key val NX or a unique-constraint insert so concurrent duplicates can’t both pass.

Example Root Cause Analysis

Finance reports a handful of orders charged twice. The audit log shows two transactions for one order milliseconds apart, and processed_events has duplicate event_id rows.

Tracing one event, two workers began processing it 3ms apart:

13:02:11.418 begin process key=evt-9f3a worker=w3
13:02:11.421 begin process key=evt-9f3a worker=w7

The queue redelivered the event (a missed ack near a visibility-timeout boundary), so two copies were in flight. The dedup code is a non-atomic check-then-act:

grep -RniE "redis.*get|redis.*set" ./src/dedup.ts

src/dedup.ts:7: const seen = await redis.get(key)
src/dedup.ts:9: if (!seen) await redis.set(key, '1')

Both workers ran GET (both saw “not seen”), then both ran SET and both proceeded to charge. The race window between GET and SET is the bug.

Fix: make the claim atomic with SET key 1 NX (or a unique constraint on event_id) so exactly one worker wins:

# Atomic claim: only the first caller gets OK, the rest get nil and skip
redis-cli SET idem:evt-9f3a 1 NX EX 86400

OK

With an atomic claim, the second worker’s SET NX returns nil and it skips the charge. Backfilling the unique constraint prevents recurrence.

Prevention Best Practices

• Treat every transport as at-least-once and make each side effect idempotent — assume retries and redeliveries will happen.
• Claim the idempotency key atomically (SET NX, INSERT ... ON CONFLICT DO NOTHING, or a unique constraint) so concurrent duplicates can’t both pass a check-then-act.
• Derive the key from something genuinely unique per operation (the event id), and propagate it unchanged across every hop.
• Set the dedup TTL longer than the maximum possible redelivery/retry window so late duplicates still hit the marker.
• Record the result alongside the key so a replay can return the original response instead of re-doing or 409ing inconsistently.
• Alert on duplicate business keys and unexpected 409 rates. The free incident assistant can correlate duplicates with the originating retry; see more automation guides.

Quick Command Reference

# Find duplicate processing
psql -c "SELECT event_id, count(*) FROM processed_events GROUP BY 1 HAVING count(*)>1 LIMIT 10;"
grep -RniE "duplicate|idempotency|409 Conflict" /var/log/*/*.log | tail

# Trace the source of the repeat
grep -RniE "<event-id>" /var/log/*/*.log | grep -iE "redeliver|retry|attempt|delivery_count"

# Verify key presence / reading
grep -RniE "Idempotency-Key|x-event-id|messageId|idempotencyKey" ./src

# Inspect dedup store and TTL
redis-cli EXISTS idem:<key>
redis-cli TTL idem:<key>

# Audit atomicity of the claim
grep -RniE "SET .* NX|ON CONFLICT|unique constraint|redis.*get|redis.*set" ./src

# Atomic claim pattern
redis-cli SET idem:<key> 1 NX EX 86400

Conclusion

An idempotency conflict or duplicate-processed event traces back to an at-least-once retry meeting an inadequate dedup guard. The usual root causes:

1. No dedup store — every redelivery re-runs the side effect.
2. The idempotency key is missing or read from the wrong field.
3. A check-then-act race lets two concurrent deliveries both proceed.
4. The key is derived from a non-unique field, causing false conflicts or false merges.
5. The dedup record TTL expired before a late retry arrived.
6. The check-and-set isn’t atomic, leaving a duplicate window.

Confirm it’s one logical event arriving twice, find the at-least-once source, then make the key claim atomic — that single change closes the window the duplicate slipped through.