Webhooks break quietly. Design for the day they do.
A webhook is a promise a third party makes to call you back. Nothing forces them to keep it.
The short version
4 things that decide this
- 01A webhook has no built-in guarantee. If the receiving endpoint is down for ten seconds, most providers retry a few times and then give up silently.
- 02Providers change payloads and deprecate API versions in release notes nobody on your team reads.
- 03A reconciliation sweep, run on a schedule, catches what the webhook missed by comparing your records against the source of truth directly.
- 04Version-pinning a webhook subscription is not optional. An unpinned integration inherits every change the provider ships, the day they ship it.
What a webhook actually promises
A webhook is an HTTP request another company's server sends to yours when something happens on their side. A payment clears, a form is filled, a shipment updates. You register a URL, and you trust them to call it every time.
That trust is the whole architecture. Your server did not ask for the event. It has no way to know one was supposed to arrive and did not. Your endpoint might be redeploying when the call comes in. A load balancer might time out. The provider's own queue can drop the job during an incident. Any of those and the event is simply gone. Most providers retry a handful of times over a few minutes or hours. Stripe gives up after roughly three days of retries with backoff. After that, nothing.
Why the failure stays invisible
A missed webhook does not throw an error in your application. Your code never runs, because the request never arrived. There is no stack trace, no failed job in a queue, nothing to alert on. The only symptom is an order that never shipped, or a subscription that never got marked cancelled, and it surfaces later as a support ticket.
- 01Payload drift. A provider adds a field, renames one, or nests data differently in a new API version. Your handler keeps parsing the old shape and either errors on a field that moved or silently ignores data that changed name.
- 02Deprecation by release note, not by code. Providers announce breaking changes in a changelog, and a team that does not read it finds out when the integration stops matching reality.
- 03A false acknowledgment. A handler that returns success before its work finishes tells the provider the event was handled, even when the database write failed right after.
Three checks that catch it
Delivery acknowledgment starts with returning the 2xx status only after the event is durably stored, not before the work is done. Log every webhook you receive, with its payload and a processed flag, before you act on it. A queue you retry from beats a handler that does everything inline and hopes.
A reconciliation sweep is the part most integrations skip. On a schedule, pull the current state directly from the provider's API and compare it against what your webhooks told you. A payment processor's list of successful charges compared against your own ledger will surface any event that never arrived, no matter why it was missed. This is the same idea as bank reconciliation, applied to an API instead of a ledger.
Version-pinning means your webhook subscription and your API calls declare a specific API version. You upgrade on your own schedule, after reading what changed. An unpinned integration is opted into every future change a provider makes, the moment they make it. Pinning turns a surprise into a scheduled task.
Questions this raises
01How do you know if you're missing webhook events?
You usually don't, until a reconciliation sweep tells you. Compare your stored records against the provider's API on a schedule, daily for high-value events like payments, weekly for lower-stakes ones. Any record present on their side and absent on yours is a missed event.
02What is a reconciliation sweep in webhook design?
A scheduled job that queries the provider's API directly and diffs the result against what your webhooks recorded. It catches drops and duplicates a webhook handler alone cannot see. The handler only knows about events that actually arrived.
03Why did our integration break after a vendor's API update?
It was probably never pinned to a version. Most providers ship breaking changes behind a version number and expect you to opt in on your own schedule. Without a pin, you inherit the change the day they release it.
