Skip to content
noughtdigital
Menu

Insights / Engineering

Webhooks that survive duplicate deliveries

Design event identity, durable processing and reconciliation so a repeated webhook does not repeat the business action behind it.

On this page

A webhook tells your application that something happened elsewhere. It does not, by itself, guarantee that your application has processed the change exactly once. A reliable integration needs to distinguish receiving a message from completing the business action behind it.

Design for the awkward sequence before choosing a queue or writing an endpoint: the provider sends an event, your application changes a record, and the acknowledgement never reaches the provider. What should happen when the message arrives again?

Give each event a durable identity

Use the provider's event identifier where its contract makes that identifier suitable. Store it with the integration account and event type so identifiers from separate accounts cannot collide accidentally. Keep a record of processing state rather than relying on an in-memory list that disappears when the process restarts.

The uniqueness check must hold under concurrent requests. Two workers can both read “not processed” before either writes “processed”. A database uniqueness constraint and an explicit claim or transaction boundary make that race part of the design instead of a rare support mystery.

For a hypothetical stock integration, the business action might be “set available quantity to the reported value”. That behaves differently from “subtract two units”. Review whether repeating the action has the same result, then protect the cases where it does not.

Verify before accepting work

Authenticate incoming events according to the provider's documented mechanism. Do not assume a request is legitimate because it contains a familiar record identifier. Preserve the original request body when signature verification requires it, and avoid logging secrets or unnecessary personal data.

Treat malformed data separately from temporary infrastructure problems. A message missing its required identifier will not improve after a hundred retries. A temporarily unavailable database might. Record a reason that an operator can understand without exposing the entire payload in a public error response.

The exact acknowledgement and retry behaviour depends on the provider. Read that contract before deciding when to return success. A fast response is only safe when accepted work has been stored somewhere that survives a process failure.

Separate delivery order from business order

Do not assume that arrival order proves which state is newest. A delayed update can arrive after a more recent one. Where the provider offers a version, sequence or trustworthy update timestamp, use the documented semantics to decide whether the message is still applicable.

Another option is to treat the event as a prompt to retrieve the current source record. That adds a dependency and may require rate-limit handling, so it is a tradeoff rather than a universal rule.

Write a reconciliation process as well. Periodically comparing important records with the source can reveal omissions that a queue dashboard never shows.

Test the failure after the side effect

A useful integration test sends the same event twice, processes two copies concurrently, and simulates a crash between the business update and the acknowledgement. Confirm that the result remains correct and that operators can tell whether the event succeeded.

Also test a permanently invalid payload and an unavailable dependency. Those paths need different recovery behaviour. The operator should be able to inspect a failed event, understand the reason and retry safely after the underlying issue is resolved.

Our integration engineering starts with these business outcomes. An endpoint returning a successful HTTP response is only the beginning of a working connection.

Keep reading

More from
the notebook.

All insights ↗

Put it into practice

Ready to build AI that actually works?

Let's discuss your AI engineering challenges and build something your users will love.