The payment webhook keeps re-sending the same notification; how do I set up idempotency?
Question
I receive webhooks from a third-party payment provider. Because of network outages, the provider may send the same successful payment notification multiple times (retries). How should I architect the webhook endpoint so my system doesn't double-process (double-spending)? How should request signing (HMAC) and DB-level idempotency key tracking work?
Answer
Short answer: these are two separate concerns — authenticity (HMAC) and preventing duplicates (idempotency). Don’t conflate them; solve each separately.
In one sentence: the signature asks “is this really the provider”, idempotency asks “have I done this work before”. Build your architecture on that split:
- Verify the signature over the RAW body. Check the provider’s HMAC signature against the raw body, before parsing, with a constant-time compare; if it doesn’t match, trust nothing and reject. Decode and re-encode the JSON and you break the signature — verify without touching the body.
- Let the DB enforce idempotency. Persist the provider’s event id (or a hash) in a table with a UNIQUE constraint; do a check-or-insert inside a transaction. The once-only guarantee comes from the database, not app logic — if the same id arrives a second time, the insert simply fails.
- Return 200 fast first, enqueue the work. Don’t run the payment side effects synchronously in the webhook handler; verify the signature, record it, return
200fast, and push the real work to a queue. Keep the worker idempotent on the same key. The response must be fast so the provider doesn’t time out and retry. - Be safe to call twice at every layer. The handler, the worker, and the side effect must each produce one clean result when called twice with the same notification. Don’t put a “it arrives once” assumption anywhere.
- Watch ordering and the replay window. A “refund” notification can arrive before its “charge”; handle out-of-order cases. Know the provider’s replay/retry window too, so you don’t mistake an old notification for a new one and process it again.
Bottom line: I’d build the flow as verify signature → return 200 fast → enqueue the work, and make the DB enforce idempotency with a UNIQUE constraint on event_id + a transactional check-or-insert. Compare the HMAC constant-time over the raw body. Double-spending is stopped by the database’s uniqueness guarantee, not by app code. You’ll find the operational detail of the queue side in the hub post.
Related Reading
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.