# Retries & ordering The retry ladder, dead deliveries, and auto-disable. A delivery succeeds on any `2xx`. Anything else — a `4xx`, a `5xx`, a redirect, a connection failure, or no answer within **10 seconds** — is a failure, and failures are retried on a fixed ladder. ## What counts as success | Response | Outcome | | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | | `200`–`299` | **Delivered.** Nothing else is read; the body is kept as a snippet for the log. | | `3xx` | Failed. Redirects are never followed — register the final URL directly. | | `4xx` | Failed, and retried. A `404` from a receiver that has not deployed yet is indistinguishable from a `404` from one that never will. | | `5xx` | Failed, and retried. | | No response in 10s | Failed, and retried. | The body of your response is never interpreted — only the status. The first kilobyte of it is stored in the delivery log so the merchant can see what your server said, with any header-shaped lines (`Set-Cookie:`, `Authorization:`) stripped before storage. ## The retry ladder An initial attempt plus eight retries, spanning roughly 45 hours: | Attempt | Sent | | ------- | -------------------------------------------- | | 1 | Immediately | | 2 | 30 seconds after attempt 1 | | 3 | 2 minutes later | | 4 | 10 minutes later | | 5 | 30 minutes later | | 6 | 2 hours later | | 7 | 6 hours later | | 8 | 12 hours later | | 9 | 24 hours later | | — | if attempt 9 fails, the delivery is **dead** | The shape is deliberate rather than exponential. The first rungs are minutes, because most failures are a deploy or a blip. The last are hours, because the rest are an outage somebody has to wake up for. Pure exponential backoff gives you neither: it is too impatient at the start and absurd at the end. Spanning \~45 hours means a Friday-evening outage is still being retried on Sunday morning. `next_retry_at` on the delivery record always says exactly when the next attempt is due, so you never have to infer your position on the ladder from this table. ## Delivery states | Status | Meaning | | ----------- | --------------------------------------------------------------------------------------- | | `pending` | Created, not yet attempted — or waiting for its next rung. | | `delivered` | A `2xx` was received. Terminal. | | `failed` | An attempt failed and **another is scheduled**. The dashboard shows this as *Retrying*. | | `dead` | The ladder is exhausted. Nothing further happens automatically. | A `dead` delivery is not lost. The event and its exact payload are kept for **30 days**, and the merchant can re-send it from **Settings → Developer → Webhooks → delivery history**. A manual retry restarts the ladder at rung one, so a receiver that has just been fixed does not wait out the 24-hour rung the automatic schedule had reached. > **Note — A disabled endpoint closes its deliveries rather than retrying them** > > If an endpoint is disabled while deliveries are queued for it, those > deliveries are marked `dead` with the reason recorded, instead of > accumulating attempts against a receiver nobody wants contacted. Turning the > endpoint back on does not resurrect them — retry the ones you want from the > log. ## Auto-disable after three days If **every** delivery to an endpoint fails continuously for three days, we turn it off. The endpoint's status becomes `auto_disabled`, and the merchant gets a dashboard notification: > **Webhook endpoint disabled** — *\* failed every delivery for 3 > days and has been turned off. Fix the receiver, then re-enable it under > Settings → Developer → Webhooks. Three things about this are worth planning around: - **It is measured from the first failure in the current failing streak**, not from a failure count. A count cannot tell "broken since Tuesday" apart from "fails its one event a month". A single success resets the clock. - **Re-enabling is manual.** An endpoint that resumed by itself would go straight back to failing, and the notification would become noise the merchant learns to ignore. - **Re-enabling clears the health verdict** — the failure streak resets, so one bad attempt after a fix does not immediately disable it again. While an endpoint is disabled, events are still recorded for the store; they are simply not delivered to that endpoint. Use `updated_after` polling to backfill whatever you missed — that is the reconciliation path webhooks never remove the need for. ## Ordering is not guaranteed It is not "usually in order" or "in order except under load". It is **not guaranteed**, and you should design as if events arrive shuffled. Deliveries run on parallel workers, each subscriber carries its own retry ladder, and a delivery that failed once is by definition behind one that did not. So `order.updated` can arrive before `order.created`; two updates to the same order can arrive in either order. Promising ordering would mean a single serialised queue per store — one slow receiver would then stall every other event for that merchant, and a permanent failure would stall them forever. That is a worse system, honestly described. **What to do instead**, and it is genuinely less work than handling order: - **Upsert, do not patch.** Every payload carries the whole object as of its own transaction, so writing it wholesale is always safe. - **Ignore arrival order; use the data.** `data.object.updated_at` and `created_at` on the envelope tell you when things actually happened. If you keep the newest snapshot per id, out-of-order delivery becomes a non-event. - **`GET` when in doubt.** The REST API is authoritative at the moment you call it. An event tells you *that* something changed; the API tells you what is true now. - **Handle the create you never saw.** If `order.updated` arrives for an id you do not know, create the record from `data.object` rather than dropping the event — the `order.created` you were waiting for may be two retries behind. > **Warning — Duplicates are not an edge case** > > A timeout on our side is indistinguishable from a slow success on yours: you > may have processed the event perfectly and lost the acknowledgement. We > retry, so you get it twice. > > De-duplicating on `X-Salaf-Event-Id` — a unique column and an > insert-or-ignore — is the entire defence, and it is three lines. See > [Handle webhooks idempotently](https://www.salafems.com/developers/guide-handle-webhooks.md). ## Retention | | | | -------------------------------- | --------------------------------------- | | Events and their payloads | **30 days** | | Delivery attempts and their logs | **30 days** (deleted with the endpoint) | After that a delivery cannot be replayed from the dashboard. If your receiver was down for longer than that, reconcile with [`updated_after`](https://www.salafems.com/developers/pagination.md#syncing-changes-with-updated_after) rather than waiting for events that no longer exist.