7 Webhook Integration Mistakes That Cause Production Outages
Avoid the most common webhook integration mistakes: missing idempotency, synchronous processing, skipping signature checks, and poor error handling that silently drops events.
Webhook integrations look simple — receive an HTTP POST, do something with the data, return 200. Most teams ship them in a day and move on. Then, three months after launch, a payment goes unacknowledged, an order hangs in limbo, or a customer gets double-charged, and the post-mortem traces back to a webhook handler built too quickly.
These are the seven mistakes that show up most often, and how to avoid each one.
Mistake 1: Processing Events Synchronously in the Request Handler
Doing your business logic — database writes, downstream API calls, email sends — inside the HTTP request handler is the fastest path to timeouts, and timeouts trigger retries.
Stripe's timeout is 30 seconds. GitHub's is 10 seconds. If your handler takes longer, the sender marks the delivery as failed and tries again. Now you are processing the same event twice, and if you have not implemented idempotency (see Mistake 2), you have a duplicate.
The fix: return 200 immediately after validating and storing the event, then hand it to a background job queue for processing.
Mistake 2: No Idempotency Handling
Webhook senders retry. Always. Network blips, your server restarting during a deploy, a temporary database timeout — any of these can cause the sender to retry an event you already received and began processing.
Without idempotency, a retried order.paid event creates a second fulfilment record. A retried subscription.created event charges the customer again.
The fix: every webhook provider assigns a stable, unique event ID. Store it in a database on first receipt. On every subsequent receipt of the same ID, return 200 and stop.
Mistake 3: Skipping Signature Verification
If you are not verifying the HMAC signature on inbound webhooks, your endpoint accepts any HTTP POST from anyone who knows its URL. A fabricated payment.succeeded event is indistinguishable from a real one without signature verification.
This is not a theoretical concern. Publicly documented webhook endpoint patterns (/webhooks/stripe, /api/shopify/callback) are scanned routinely.
The fix: implement HMAC-SHA256 verification using the raw request body before doing anything else with the payload. Return 401 on any signature mismatch.
Mistake 4: Using the Wrong HTTP Status Codes
Returning 500 when you should return 200, or 200 when you should return 4xx, teaches the sender the wrong thing about what to retry.
A 500 tells the sender your endpoint is having a transient problem — retry later. If you return 500 for a bug in your parsing logic, the sender will keep retrying indefinitely, flooding your endpoint with events that will always fail.
A 200 for a duplicate event that you silently discarded is correct. A 200 for an event you actually failed to process is wrong — it tells the sender you succeeded when you did not.
The fix: return 200/202 only when you have successfully received and queued the event. Return 5xx for genuine transient failures. Return 4xx sparingly and intentionally, knowing the sender may not retry.
Mistake 5: Trusting Webhook Order
Webhook providers do not guarantee delivery order. Under retry conditions, it is entirely possible to receive subscription.updated before subscription.created. Most teams assume events arrive in chronological order and build logic that breaks when they do not.
The fix: design handlers to be order-tolerant. Use the event's embedded timestamp or sequence number (if the provider includes one) to decide whether to apply an update. Use upserts rather than inserts where possible.
Mistake 6: No Dead-Letter Strategy
Providers retry for a finite window — Stripe for 3 days, Shopify for about 4 hours. Once that window closes, the event is gone from the provider's side. If your endpoint was down for the entire retry window, or if processing failed every single attempt, you have lost the event.
Most teams discover this when a customer reports a missing record days later and the investigation reveals there is no way to replay it.
The fix: build a dead-letter store. Any event that exhausts your internal retry attempts gets written to a dead_letter_events table with the full payload. Build a replay mechanism so you can reprocess events after you fix the underlying bug.
Mistake 7: Hardcoding Secrets in Code
Webhook signing secrets committed to a repository expose you to two risks: anyone with repo access can forge events, and rotating the secret requires a code change and a deploy.
The fix: load signing secrets from environment variables or a secrets manager. Verify that your CI pipeline does not log environment variables. Ensure the secret rotation path is tested and fast.
A Common Pattern in All of These
Each of these mistakes is a version of the same root cause: treating the webhook endpoint as a simple HTTP handler rather than as the entry point to an event-driven pipeline that needs the same reliability guarantees as any other critical data path.
If you are building or auditing webhook integrations for a product that handles real customer data or financial transactions, Clixo can help you design and review the reliability layer.