In this guide 6 sections
Senders retry any webhook that does not get a quick success reply, so duplicates and delays are normal. Network paths also reorder events. Build the receiver to store each event first, verify its signature, process it later, and read the current status from the sender’s API before changing a record.
Delivery is at least once, never exactly once
A webhook is an HTTP request fired by the sender when something happens. If your server answers with anything but a quick 200, the sender assumes failure and tries again. Retries run for hours, sometimes days.
So duplicates are not a bug in the sender. They are the promise. The sender would rather deliver an event twice than lose it, and it leaves the job of ignoring repeats to you.
Think of the sender as a courier who rings the bell until someone signs, and who rings again if your signature takes too long to appear.
Why events show up late or in the wrong order
Lateness has dull causes. Your endpoint timed out while a slow database query ran, a deploy restarted the server, or a firewall rule dropped the request. The retry then arrives long after the event.
Order breaks because each event travels on its own. An order.updated event can overtake the order.created that should precede it, especially when retries are involved. A cancelled status can then be overwritten by an older paid status.
Load makes it worse. A burst of retried events lands beside fresh ones, and a handler without a queue processes them in whatever sequence the threads happen to finish.
- Receiver took longer than the sender’s timeout
- Server restarted during a deploy
- Sender retried with backoff gaps
- Two events took different network paths
- Queue workers handled events in parallel
Store first, process later
Do the least possible inside the request. Check the signature header against the raw body, write the event ID and payload to an events table, and return 200. Everything else happens in a worker.
The table gives you a unique key on the event ID, so a repeat is rejected by the database itself. It also gives you a log. When a customer asks why an invoice never updated, you can see whether the event arrived.
- Verify the HMAC signature on the raw body
- Reject timestamps older than a few minutes
- Insert with a unique event ID
- Return 200 before any heavy work
- Keep failed events for replay
Check status instead of trusting the payload
The payload is a notification, not the source of truth. When the worker picks up a payment.captured event, it should call the sender’s API, fetch the payment by ID and use the status it returns now.
This one habit fixes ordering. A stale event for a refunded payment simply reads the refunded status and changes nothing. Where the sender gives a version number or an updated timestamp, store it and skip anything older than what you hold.
Test the ugly cases on purpose
Replay the same event five times and confirm that only one invoice changes. Send an update before its creation event. Stop your worker for an hour, then start it and watch the backlog drain without duplicates.
Most providers offer a test mode or a replay button for exactly this. Run these three drills once, write down the results and repeat them after every major change to the handler.
Keep a short runbook beside the code. Whoever is on call at midnight should know where the events table lives, how to replay a day, and who holds the dashboard login.
Webhook receiver checklist
A one-page checklist in two columns. The left column lists receiver rules: signature check, timestamp window, unique event ID, fast 200, worker queue, status fetch. The right column holds a pass or fail box and a note line for each. Below sits a short table for the three drills: duplicate, out of order and outage backlog.
Open a blank worksheet to printQuestions owners ask
How fast must my endpoint reply?
Reply within a few seconds, ideally under one. Check the provider’s documented timeout, because each one differs. Do the real work after you have stored the event.
Can I rely on event order if I use a queue?
No. A queue keeps order only for events it received in that order, and the sender may already have sent them shuffled. Compare version fields or fetch current status instead.
How do I know a webhook is genuine?
Compute the signature over the raw request body with your shared secret and compare it with the header. Reject anything that fails. Parsing the JSON first can alter the bytes and break the match.
What if my server was down for a day?
Most senders keep retrying for a while and let you replay missed events from a dashboard. After an outage, replay the gap and rely on your unique event ID to skip what you already hold.