Skip to main content

Webhook deliveries failing

A webhook endpoint stops receiving events: the retry schedule, the automatic circuit breaker that pauses a failing consumer, and how deliveries recover.

A registered webhook endpoint has stopped receiving some or all of its events. Open its delivery history: its shape tells you whether attempts keep failing, have stopped, or succeeded while your consumer disagrees. A webhook endpoint is a URL you register with a DALP instance so the platform pushes a signed HTTP request to it whenever a relevant event happens. It covers outbound webhook delivery from the platform to your consumer. Webhooks your instance receives from compliance providers are a different surface with their own management screens.

Every check reads from the endpoint's delivery history: one row per event with its attempts nested under it, each attempt with its status, its failure class, and the response your consumer returned. In Console, open Organisation settings, then Webhooks, then the endpoint. The Event deliveries tab lists every event with its attempts, and clicking a row opens the Webhook event sheet with the failure class, the signed request headers, the response, and a ready-made repro command. The same history is available from the dalp command-line tool as dalp get webhooks deliveries <webhookId>, and over the webhook management endpoints.

Whatever the cause, the history shows one of three shapes: attempts that keep failing, attempts that have stopped entirely, or attempts that succeeded while your consumer disagrees. Breaker, Reactivate, replay and the address policy are on the platform side. DNS, TLS, response codes, the signing secret and receipts are on the consumer side, so send the failure class to the consumer's owner.

Symptoms

  • A registered webhook endpoint stops receiving events it previously received.
  • Deliveries to one endpoint fail while other endpoints on the same instance keep receiving events normally.
  • The endpoint's status in Console changed to Auto-disabled without anyone switching it off.
  • Your consumer reports missing events for a time window, then receiving them all at once, or not at all.

Checks

Read the delivery history first. Its shape tells you which check to start with.

The delivery history's shape picks the check to run.Start with the Event deliveries tab, which lists one row per event with its attempts, each with its failure class and the consumer's response. Three shapes follow. Failing rows that keep appearing split on the failure class: a retryable class such as DNS, TLS, a timeout, or a 5xx keeps retrying for three days, so fix reachability under check 3. A terminal 4xx is permanent for that one event, handled under check 4. No new rows at all means the circuit breaker is open: go to check 2. Rows that show delivered while your consumer disagrees point at receipt deadlines and duplicate handling under check 5.The delivery history's shape picks the check to run.The same decision tree as the desktop variant, drawn portrait, with the same three top-level shapes. Failing rows that keep appearing split further on the failure class: a retryable class such as DNS, TLS, a timeout, or a 5xx keeps retrying for three days, so fix reachability under check 3. A terminal 4xx is permanent for that event, handled under check 4. No new rows at all means the breaker is open, go to check 2. Delivered rows the consumer disputes point at receipt deadlines or duplicates, go to check 5.

The delivery history sorts a failing endpoint into three shapes: attempts that keep failing, attempts that stopped, and deliveries the consumer disputes.

  1. Confirm whether the endpoint's failures are transient or sustained. The signal is the Event deliveries tab: a healthy endpoint shows mostly delivered rows, with occasional failed attempts followed by a delivered one for the same event. A single failed attempt retries automatically on an exponential schedule that starts near 30 seconds and roughly doubles after each attempt, with random jitter so many endpoints failing at once do not retry in lockstep. Retries stop 3 days after the platform recorded the event. The next attempt after that records the abandonment, and an open breaker delays that attempt, so the entry can appear after the event has aged out, as check 2 explains. A handful of retries in the history is normal. Some failures never retry: a delivery your consumer rejects with an ordinary 4xx response, other than a request timeout or a rate-limit signal, is permanent for that event. A fix made afterward recovers future events but never that one. Each attempt itself times out after 20 seconds if the consumer never responds.

  2. Check whether the platform has stopped attempting deliveries. The signal is the Breaker field under Endpoint details on the endpoint's Overview tab. “Closed, deliveries flowing” means deliveries flow. “Open, deliveries paused”, with the endpoint shown as Auto-disabled, means the breaker tripped and the platform is skipping deliveries to this endpoint rather than attempting them.

    • Tripping. An endpoint trips the breaker after 30 consecutive failed deliveries. Missed receipt deadlines count toward that streak, manually sent test events never do. Events that happen while the breaker is open are not queued for the endpoint, so neither recovery nor a replay delivers them later.
    • Recovery. Recovery is automatic: the platform probes an open endpoint by re-sending its most recent live event, each probe at least a minute apart with 10 seconds to answer. A failed probe resets the success count, so a consumer that flaps between working and failing stays paused until it answers twice in a row. Two consecutive probe successes move the endpoint to Recovering, and live deliveries resume immediately. The next delivered event returns it to Enabled.
    • The Recovering trap. Entering Recovering does not reset the failure streak that tripped the breaker in the first place. One failed live delivery there reopens the breaker immediately, before a second probe or a delivered event ever gets the chance. A consumer that is only half fixed can loop through probe, Recovering, and Auto-disabled without a single live delivery landing.

  3. Verify the consumer's endpoint is reachable and answering correctly. The signal is the failure class in the Last response column on each row.

    • Reading the taxonomy. It separates the ways a request can die: the name never resolved, the TLS handshake failed, the connection never opened, the response never finished, or the consumer answered with an error status. The sixth value is an unparseable response. A request the platform refuses under the two address rules that follow never reaches the consumer and records TARGET_REFUSED. The healthy value is a delivered row, which the platform records only for a 2xx response.
    • Network failures. All of them retry on the normal schedule, so fixing the consumer is enough and no platform-side change is needed.
    • Platform policy. The platform checks endpoint address and HTTPS policy at registration and before each dispatch. A URL that fails those checks cannot be registered. The endpoint URL must resolve to a publicly routable address. A receiver on private address space fails that check until the operator adds its range to global.dalpApp.config.webhooks.allowedTargetCidrs in the deployment's Helm values. The URL must also use https. Both rules share one exception: an origin the operator lists in DALP_WEBHOOKS_TRUSTED_DESTINATIONS, for a private-network install, is exempt from the https requirement and the public-address check alike, so an endpoint on that list can use http:// or a private address. Both settings widen the delivery policy for every organisation on the instance, not only the endpoint you are fixing, and a trusted origin receives deliveries over plain HTTP. List only private receivers your own team runs. Never list a DALP origin or another internal service whose responses must stay private, because the delivery history records the response, so a webhook manager in any organisation could read it through a delivery. Tenant-facing APIs cannot add an origin to that list. Only an operator changing the deployment's configuration can. An attempt that fails either rule before dispatch sends nothing, records TARGET_REFUSED, and does not retry: correct the endpoint URL or the deployment's configuration, then use Resend for each affected event.

  4. If the integration verifies signed payloads, confirm the consumer holds the current signing secret. The signal is a run of terminal 4xx rows that started when a secret changed on either side. A consumer that rejects a signature usually does so with an ordinary 4xx, which the platform treats as permanent for that event. Fixing the secret recovers deliveries from that point forward but does not restore the events rejected while it was wrong. Replay them, as Actions describes. Two behaviours keep rotation itself from causing this. After a rotation the previous secret stays valid for 24 hours, and the platform signs with both secrets during the overlap. An event that waited in retry backoff is re-signed with a fresh timestamp before the next attempt, inside the Standard Webhooks tolerance window. A long network delay or clock mismatch can still exceed the receiver's timestamp tolerance. Check receiver time and the verified delivery headers.

  5. If the endpoint requires counter-signed receipts, confirm the consumer responds inside its receipt window. That window is 30 seconds by default. Receipt timeout in the endpoint's Edit sheet, or receiptTimeoutSeconds through the API, sets it to a whole number of seconds from 1 to 60. The signal is rows that reached the consumer but still count as failures because the receipt never arrived in time. A late receipt is treated like any other failed delivery and retried on the normal schedule. A receipt whose signature does not verify is refused without changing the delivery, so the consumer can sign it again inside the window. A correctly signed receipt whose body hash does not match is different: it ends delivery of the event, on the same footing as a terminal 4xx. A corrected receipt for the same delivery still restores the attempt as delivered.

The circuit breaker recovers automatically, and one failed delivery during recovery reopens it at once.A healthy endpoint delivers normally while individual failures retry on the backoff curve. Thirty consecutive failed deliveries trip the breaker, and test events never count. While the breaker is open the platform skips deliveries to the endpoint rather than attempting them, and Console shows the endpoint as Auto-disabled. Test events still deliver and change nothing. The platform probes the open endpoint automatically by re-sending its most recent live event, each probe at least a minute apart with ten seconds to answer, and a failed probe resets the success count. Two consecutive probe successes move the endpoint to Recovering, where live deliveries resume immediately, and from there two things can happen. The failure count that tripped the breaker carries over unchanged: one failed live delivery while Recovering reopens the breaker at once, looping back to Open before a second probe or a delivered event ever gets the chance. Only if a live delivery succeeds first does the endpoint return to Enabled, with the failure streak reset to zero. A manual shortcut exists: the Reactivate action on the endpoint's Overview tab re-enables the endpoint and closes the breaker at once.The circuit breaker recovers automatically, and one failed delivery during recovery reopens it at once.The same lifecycle as the desktop variant, drawn portrait. Thirty consecutive failed deliveries trip the breaker, with test events never counting. While open, deliveries are skipped and Console shows Auto-disabled, though test events still deliver. Automatic probes re-send the latest live event at least a minute apart, and a failed probe resets the count. Two consecutive successes move the endpoint to Recovering, where live deliveries resume, and from there two things can happen. The failure count that tripped the breaker carries over: one failed live delivery there reopens the breaker at once, looping back to Open. Only a live delivery that succeeds first closes the breaker fully. Reactivate on the Overview tab closes it at once.

The breaker trips after 30 consecutive failed deliveries, recovers through probes, and reopens at once if one live delivery fails while Recovering.

Actions

  • If the breaker tripped, fix the underlying reachability or response problem first. The platform resumes on its own once its probes confirm recovery: two consecutive successes, each at least a minute apart, so recovery takes at least two minutes after your fix, and longer when a probe fails. Events raised during that wait are not delivered to this endpoint later, so once the fix is in place, prefer to recover at once: on the endpoint's Overview tab, Reactivate, or Reactivate and replay failed events when failed events are loaded, re-enables the endpoint and closes the breaker at once. Enable in the endpoint's actions menu does the same. Use Send test event to confirm your fix either way. A test event delivers even while the breaker is open and never counts toward tripping or clearing it, so it confirms your fix without changing the breaker.
  • If events aged past the 3-day retry window, or failed terminally, replay them. Replay covers only events the platform recorded for this endpoint. Events raised while the breaker was open were never recorded for it, so no replay returns them: reconcile that window from the API, from the time the breaker opened until it closed. The webhook replay call re-queues either one event by its id or every matching event in a block range. Each replayed delivery is flagged as a replay in its envelope, so your consumer can tell it from a first delivery. Replay is refused while the endpoint is disabled or the breaker is open, so recover the endpoint first, then replay. Check the delivery history for events approaching the 3-day window rather than waiting for an alert.
  • If signature verification was the failure, correct the consumer's secret, then replay. A rejected signature is permanent per event, so the events that failed while the secret was wrong stay failed. The same replay and retry calls cover both a window of them and a single delivery you want to re-queue on its own. Deliveries themselves are otherwise working, so no platform-side change is needed.
  • If your consumer sees a duplicate event, it is normal at-least-once delivery within the 3-day retry window. The platform journals delivery attempts, and a retry reuses the same event id as its original delivery throughout. Deduplicate retries on the webhook-id header, which carries that id on every attempt. A replay is different: it mints a fresh event id for each event it re-queues, so the webhook-id header on a replayed delivery does not match the original and deduplicating on it does not work. Use is_replay to recognise the replay, then use stable domain identifiers from the event payload to avoid applying the same outcome twice.

Verify recovery

New events show as delivered in Event deliveries, the endpoint's Breaker field reads “Closed, deliveries flowing”, and replayed events reach your consumer flagged as replays.

Escalate

Escalate to your DALP support contact with the endpoint id, the failure window, whether the breaker had tripped, and the failure class shown in the delivery history. That field distinguishes a consumer-side outage from a platform delivery defect. A name that never resolved points at DNS or egress, a terminal 4xx points at the consumer's validation, and a run of timeouts points at the consumer's capacity.

Frequently asked questions

Does the 3-day retry window keep counting while the breaker is open?

Yes. The window is measured from when the platform recorded the event, and it does not pause while the breaker skips deliveries. A consumer that stays broken for days can therefore age events out without a single further attempt appearing in the history. After a long outage, replay the events that aged out before the breaker tripped, and reconcile from the API the window the breaker was open, which no replay covers.

What happens to a delivery when the platform's own database fails part-way through it?

The attempt pauses and the platform runs it again later. The first rerun comes about a minute after the failure, and each further wait is twice as long as the one before, up to six hours. The endpoint's later events wait behind the paused one, so they still arrive in order once it goes through. Each rerun signs the event again before it posts, so a delivery sent after a long pause still carries a fresh signature. Your consumer can receive the same delivery twice when the first post went out before the failure. Deduplicate on the webhook-id header as for any retry.

If the attempt still fails after ten reruns, about eight and a half hours after the first, or cannot be completed at all, the platform marks that event as failed with the reason attempt_workflow_died and logs the error. The endpoint's later events then keep flowing. Replay the failed event with the webhook replay call.

Does rotating the signing secret break deliveries already in flight?

No, provided the consumer finishes its switch inside the 24-hour overlap described in check 4. After the overlap ends, the old secret stops signing, and a consumer still pinned to it starts rejecting deliveries as terminal 4xx failures, the same signature scenario check 4 covers.

Was this page helpful?