SALAF EMSDevelopers
salafems.com

Retries & ordering

The retry ladder, dead deliveries, and auto-disable.

A delivery succeeds on any 2xx. Anything else — a 4xx, a 5xx, a redirect, a connection failure, or no answer within 10 seconds — is a failure, and failures are retried on a fixed ladder.

What counts as success

ResponseOutcome
200299Delivered. Nothing else is read; the body is kept as a snippet for the log.
3xxFailed. Redirects are never followed — register the final URL directly.
4xxFailed, and retried. A 404 from a receiver that has not deployed yet is indistinguishable from a 404 from one that never will.
5xxFailed, and retried.
No response in 10sFailed, and retried.

The body of your response is never interpreted — only the status. The first kilobyte of it is stored in the delivery log so the merchant can see what your server said, with any header-shaped lines (Set-Cookie:, Authorization:) stripped before storage.

The retry ladder

An initial attempt plus eight retries, spanning roughly 45 hours:

AttemptSent
1Immediately
230 seconds after attempt 1
32 minutes later
410 minutes later
530 minutes later
62 hours later
76 hours later
812 hours later
924 hours later
if attempt 9 fails, the delivery is dead

The shape is deliberate rather than exponential. The first rungs are minutes, because most failures are a deploy or a blip. The last are hours, because the rest are an outage somebody has to wake up for. Pure exponential backoff gives you neither: it is too impatient at the start and absurd at the end. Spanning ~45 hours means a Friday-evening outage is still being retried on Sunday morning.

next_retry_at on the delivery record always says exactly when the next attempt is due, so you never have to infer your position on the ladder from this table.

Delivery states

StatusMeaning
pendingCreated, not yet attempted — or waiting for its next rung.
deliveredA 2xx was received. Terminal.
failedAn attempt failed and another is scheduled. The dashboard shows this as Retrying.
deadThe ladder is exhausted. Nothing further happens automatically.

A dead delivery is not lost. The event and its exact payload are kept for 30 days, and the merchant can re-send it from Settings → Developer → Webhooks → delivery history. A manual retry restarts the ladder at rung one, so a receiver that has just been fixed does not wait out the 24-hour rung the automatic schedule had reached.

Auto-disable after three days

If every delivery to an endpoint fails continuously for three days, we turn it off. The endpoint's status becomes auto_disabled, and the merchant gets a dashboard notification:

Webhook endpoint disabled<name> failed every delivery for 3 days and has been turned off. Fix the receiver, then re-enable it under Settings → Developer → Webhooks.

Three things about this are worth planning around:

  • It is measured from the first failure in the current failing streak, not from a failure count. A count cannot tell "broken since Tuesday" apart from "fails its one event a month". A single success resets the clock.
  • Re-enabling is manual. An endpoint that resumed by itself would go straight back to failing, and the notification would become noise the merchant learns to ignore.
  • Re-enabling clears the health verdict — the failure streak resets, so one bad attempt after a fix does not immediately disable it again.

While an endpoint is disabled, events are still recorded for the store; they are simply not delivered to that endpoint. Use updated_after polling to backfill whatever you missed — that is the reconciliation path webhooks never remove the need for.

Ordering is not guaranteed

It is not "usually in order" or "in order except under load". It is not guaranteed, and you should design as if events arrive shuffled.

Deliveries run on parallel workers, each subscriber carries its own retry ladder, and a delivery that failed once is by definition behind one that did not. So order.updated can arrive before order.created; two updates to the same order can arrive in either order.

Promising ordering would mean a single serialised queue per store — one slow receiver would then stall every other event for that merchant, and a permanent failure would stall them forever. That is a worse system, honestly described.

What to do instead, and it is genuinely less work than handling order:

  • Upsert, do not patch. Every payload carries the whole object as of its own transaction, so writing it wholesale is always safe.
  • Ignore arrival order; use the data. data.object.updated_at and created_at on the envelope tell you when things actually happened. If you keep the newest snapshot per id, out-of-order delivery becomes a non-event.
  • GET when in doubt. The REST API is authoritative at the moment you call it. An event tells you that something changed; the API tells you what is true now.
  • Handle the create you never saw. If order.updated arrives for an id you do not know, create the record from data.object rather than dropping the event — the order.created you were waiting for may be two retries behind.

Retention

Events and their payloads30 days
Delivery attempts and their logs30 days (deleted with the endpoint)

After that a delivery cannot be replayed from the dashboard. If your receiver was down for longer than that, reconcile with updated_after rather than waiting for events that no longer exist.