Webhook Idempotency at Scale: Redis Locks, Queues, and the HTTP 202 Pattern

Any system that receives webhooks — payment providers, ad platforms, messaging APIs — will eventually receive the same delivery more than once, because the sender's retry policy assumes failure whenever it doesn't get a fast response. The fix isn't to write handlers that hope duplicates never happen. It's to acknowledge before you process, deduplicate atomically before you acknowledge to your own workers, and write every handler as if it will run more than once anyway, because eventually it will.

Retries happen for reasons entirely outside your control: a network blip between the sender and your load balancer, a slow database write that trips the sender's own timeout, or a scheduled redelivery job on the sender's side that has no idea you already processed the first attempt. Processing a webhook inline, inside the same request that acknowledges it, is exactly the pattern that turns a slow response into a duplicate delivery — the sender times out waiting for your 200, assumes failure, and retries while your first request is still running.

The Three-Stage Pipeline: Verify, Deduplicate, Enqueue

Stage 1 — Verify the Signature First

Order matters here. Deduplication has to happen after signature verification, never before — otherwise an unauthenticated request with a guessed or garbage delivery ID could occupy a lock key and cause a legitimate delivery with the same ID to be silently dropped.

Stage 2 — Claim the Delivery With an Atomic Lock

Once the signature checks out, claim the delivery ID with a single atomic Redis command. `SET NX EX` sets the key only if it doesn't already exist and attaches an expiry in the same call, so there's no race between checking for existence and setting the key — a check-then-set done as two separate commands is exactly the kind of race condition this whole pipeline exists to avoid.

Stage 3 — Acknowledge Fast, Process Async

The HTTP handler returns immediately after the lock is claimed and the job is enqueued — a 202 Accepted, not a 200 that waits for the actual processing to finish. The real work happens in a worker that can retry independently of the original request.

Wrong vs Right: Processing Inline vs Handing Off to a Queue

The anti-pattern is subtle because it works fine at low volume: the handler verifies the signature and then does the real work — writing to the database, calling another service — before responding. It reads simply and passes every manual test. Under load, or the moment a downstream call gets slow, it's exactly what causes the sender's own timeout to fire and trigger the retry that creates the duplicate this whole pipeline is meant to prevent.

Handling Out-of-Order Delivery

Deduplication solves for the same event arriving twice. It doesn't solve for two different events about the same resource arriving out of order — a sender's own retry queue or load balancing can deliver an "order updated" event after a later "order cancelled" event that was actually sent first. If your handler blindly applies whatever arrives last, you can end up applying an older state on top of a newer one. The fix is to carry a sequence number or timestamp in the payload and have the handler compare it against the last-applied value for that resource before writing, discarding anything older instead of assuming delivery order matches send order.

Rotating Webhook Secrets Without Downtime

The signing secret itself needs to be rotatable without a coordinated flag day, because the sender and receiver rarely deploy at the same moment. Accept two active secrets during a rotation window — verify against the new one first, fall back to the old one, and only remove the old secret once you've confirmed the sender has switched. Treat the secret the same way you'd treat a database credential: stored in a secrets manager, not committed anywhere, and rotated on a schedule rather than only when something goes wrong.

Guarding Against Replay of Old, Validly-Signed Requests

A signature proves the payload wasn't tampered with. It doesn't prove the request is fresh — a captured, validly-signed request can be replayed later by anyone who intercepted it. Most webhook providers include a timestamp header alongside the signature specifically so receivers can reject anything too old, even if the signature itself still checks out.

Why Redis Alone Isn't Enough

Redis can restart, evict keys under memory pressure, or fail over mid-lock. Treat the lock as a fast, cheap optimization that stops most duplicates at the door — not as your actual correctness guarantee. The real guarantee belongs to the database: a unique constraint on the delivery ID and an upsert that no-ops on conflict.

Picking Lock TTLs Without Guessing

Too short a TTL and a duplicate can slip through while the first attempt is still processing. Too long, and a legitimate retry after a real transient failure gets silently dropped for no good reason. A two-key pattern separates these concerns cleanly: a short-lived "processing" lock that guards against near-simultaneous duplicates, and a long-lived "done" record that guards against much later redeliveries.

Backpressure and the Dead Letter Queue

Monitor queue depth, not just error rate — a growing backlog is often the earlier warning sign. Jobs that exhaust their retry budget should land in a dead letter queue rather than disappear; once you've fixed whatever caused the failures, replay tooling for the DLQ lets you reprocess those deliveries deliberately instead of hoping the sender retries again on its own schedule.

What to Actually Alert On

Three numbers matter more than raw error counts for this kind of pipeline. Queue depth over time tells you whether workers are keeping up or falling behind. The rate of "duplicate discarded at the gate" log lines tells you whether a sender has started retrying more aggressively than usual, which is often the first sign something downstream got slow. And DLQ growth tells you whether a category of job is failing systematically rather than transiently — a single job failing five times and landing in the DLQ is normal; twenty unrelated jobs doing the same thing in the same minute is an incident.

What This Looks Like at Vasiva

Dima ingests webhook and callback traffic from several external sources — ad platforms and messaging channels among them — into its unified inbox and performance views. The shape is the same lock-then-queue pipeline described here: verify, claim, enqueue, and let a database constraint be the final word on whether something has already been recorded.

If duplicate writes, timeout-driven retries, or a webhook pipeline that's starting to buckle under load sound familiar, that's the kind of problem our Product Engineering and Internal Tools work is built around. Reach out if you want a second opinion on your event pipeline.