Webhook Debugging FAQ: How to Trace and Replay Failed Webhook Deliveries

A webhook integration stops working and you have no idea what happened. The customer says the order did not process. The provider's dashboard shows the event was sent. Your logs show nothing. Debugging webhook failures is harder than debugging a synchronous API call — there is no stack trace, no error response visible to the client, and the failure may have happened hours ago in a background job.

This FAQ covers the practical questions engineers face when webhook deliveries go wrong.

Why is the provider showing "delivered" but my system did nothing?

"Delivered" from the provider's perspective means your endpoint returned 2xx. It does not mean your business logic ran successfully — only that the HTTP response was 200. Common causes:

• Your endpoint returned 200 immediately and the background job failed silently.
• The background job succeeded but wrote to the wrong database or a feature flag disabled processing.
• Your deduplication check marked the event as already processed, but it was not.

Check: look at your background job logs for the event ID specifically. Search your dead-letter table for the event ID. If the dedup store shows "seen" but no processing occurred, you have a bug in the dedup-to-enqueue path.

How do I trace a specific event end-to-end?

Every provider's event has a stable unique ID. Build your entire logging pipeline around this ID from the moment the event arrives:

• Log the event ID when the webhook is received and validated.
• Log the event ID when the background job starts.
• Log the event ID with the outcome (success or failure) when the job completes.

Use a structured logging format (JSON lines) so you can query by event ID across all services. In a distributed system, pass the event ID as a correlation header to downstream services.

If you have this in place, finding a specific event means running:

grep '"event_id":"evt_abc123"' /var/log/app/*.log

Or the equivalent query in your logging platform (Datadog, Grafana Loki, CloudWatch Logs Insights).

How do I replay a failed event?

Using the provider's dashboard

Most providers let you manually trigger a re-delivery from their dashboard. Stripe has a "Resend" button on individual events. GitHub has a "Redeliver" button. This is the fastest path for a one-off failure.

Be aware: if your system already processed the event (the provider shows delivered, your code ran, but produced wrong output), replaying will run your handler again. Make sure your handler is idempotent before replaying.

Using your own dead-letter queue

If you have a dead-letter table, build a replay script that:

1. Fetches events from the dead-letter table matching a filter (event type, time range, source).
2. For each event, re-enqueues it to the normal processing queue.
3. Marks the dead-letter record as replayed with a timestamp.

This is safer than replaying from the provider because you control the replay rate and can test against staging first.

My endpoint is returning 200 but the provider keeps retrying. Why?

The provider is calling your endpoint, you are returning 200, but their dashboard shows continued retries. This usually means:

• Your endpoint is returning 200 for the wrong route. Verify the URL registered in the provider's dashboard exactly matches your deployed endpoint path, including trailing slashes.
• DNS or routing is sending the requests somewhere else. Test by having the provider send a test event and checking your actual access logs — not application logs, access logs — to confirm the POST hit your server.
• The provider is rotating through multiple endpoints if you registered more than one.

A quick way to test: use a tool like https://webhook.site as a temporary URL to confirm the provider is actually sending events to the correct URL and with the correct payload.

How do I test webhook handlers without waiting for real events?

Use the provider's test event feature. Stripe, Shopify, and GitHub all support sending synthetic test events from their dashboards or CLIs. Stripe's CLI (stripe trigger payment_intent.succeeded) fires a real-looking event with a test payload.

Set up a local tunnel. Tools like ngrok, Cloudflare Tunnel, or the Stripe CLI's built-in forwarding create a public URL that tunnels to your local development server. You can iterate and debug without deploying.

Write unit tests against your handler logic directly. Separate your business logic from the HTTP handler so you can test it with a hardcoded payload without any HTTP layer involved. This is the fastest feedback loop for debugging processing bugs.

What observability should I have before going to production?

At minimum:

• A counter for events received, broken down by event type.
• A counter for events that failed signature verification.
• A counter for background job failures, broken down by event type.
• Dead-letter queue depth as a gauge metric — alert when it grows.
• A p95 and p99 latency metric for background job processing time.

Add alerting on: error rate exceeding a threshold, dead-letter depth exceeding a threshold, and processing latency exceeding your SLA. These three alerts will catch the vast majority of production webhook problems before a customer notices.

How do I know if I am missing events?

Build a reconciliation job. For most integrations, you can query the provider's API for a list of recent events and compare them against your local records. Any event present in the provider's list but absent in your database is a missed event.

Stripe's Events API (/v1/events) supports time-range filtering. Run this comparison daily or hourly. This catches events lost during provider retry window expiration, network-level losses, or bugs in your dedup logic.

Getting webhook observability right before you need it saves hours of debugging after you do. If you are building or auditing a webhook integration, Clixo can review your architecture and instrumentation.