# Handle webhooks idempotently Verify, dedupe, upsert, and answer fast. A correct webhook receiver is four steps in a fixed order. Get the order wrong and you have a receiver that works in testing and corrupts data in production. ```text 1. verify the signature ─ before parsing, against the raw bytes 2. de-duplicate on event id ─ delivery is at-least-once 3. answer 2xx ─ within 10 seconds, before doing the work 4. upsert from data.object ─ off the request, in any order ``` ## The whole receiver **Node.js** ```javascript app.post( "/webhooks/salaf", express.raw({ type: "application/json" }), async (req, res) => { // 1. Verify BEFORE parsing or trusting anything. const ok = verifySalafSignature( req.body, req.get("X-Salaf-Signature"), process.env.SALAF_WEBHOOK_SECRET, ); if (!ok) { return res.sendStatus(400); // we will not retry a 4xx into working } const event = JSON.parse(req.body.toString("utf8")); const eventId = req.get("X-Salaf-Event-Id"); // === event.id // 2. Dedupe. Delivery is at-least-once: the same event WILL arrive // twice eventually. A unique index on the event id is the whole trick. const inserted = await db.query( `INSERT INTO salaf_events (id, type, payload) VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING`, [eventId, event.type, event], ); // 3. Answer FAST. Anything slower than a few hundred ms risks our 10s // timeout, and a timeout means a retry you did not need. res.sendStatus(200); if (inserted.rowCount === 0) { return; // already handled — nothing more to do } // 4. Do the work off the request. Order is NOT guaranteed, so write // upserts: order.updated may land before order.created. void queue.add("salaf-event", { eventId }); }, ); async function handleEvent(event) { switch (event.type) { case "order.created": case "order.updated": case "order.paid": // data.object is the full snapshot as it was inside the transaction // that produced it — upsert it, do not patch field by field. return upsertOrder(event.data.object); case "inventory.updated": return setLevels(event.data.object.variant_id, event.data.object.levels); default: return; // unknown types are normal: the catalog only ever grows } } ``` **Python** ```python import json from django.db import IntegrityError from django.http import HttpResponse @csrf_exempt @require_POST def salaf_webhook(request): # 1. Verify before trusting the body. if not verify_salaf_signature( request.body, request.headers.get("X-Salaf-Signature"), settings.SALAF_WEBHOOK_SECRET, ): return HttpResponse(status=400) event = json.loads(request.body) event_id = request.headers["X-Salaf-Event-Id"] # same as event["id"] # 2. Dedupe on the event id — delivery is at-least-once. try: SalafEvent.objects.create( id=event_id, type=event["type"], payload=event ) except IntegrityError: return HttpResponse(status=200) # already seen; still a success # 3. Queue the work and answer immediately: our timeout is 10 seconds, # and a slow 200 costs you a duplicate you did not need. process_salaf_event.delay(event_id) return HttpResponse(status=200) def handle_event(event): kind = event["type"] obj = event["data"]["object"] # 4. Upsert, never patch: events can arrive out of order, and every # payload carries the complete object as of its own transaction. if kind in ("order.created", "order.updated", "order.paid"): upsert_order(obj) elif kind == "inventory.updated": set_levels(obj["variant_id"], obj["levels"]) # Unknown types are expected — the catalog only ever grows. ``` **PHP** ```php getContent(), $request->header('X-Salaf-Signature'), config('services.salaf.webhook_secret'), )) { return response()->noContent(400); } $event = json_decode($request->getContent(), true); $eventId = $request->header('X-Salaf-Event-Id'); // same as $event['id'] // 2. Dedupe on the event id (unique column). At-least-once delivery // means duplicates are routine, not exceptional. $isNew = SalafEvent::query()->insertOrIgnore([ 'id' => $eventId, 'type' => $event['type'], 'payload' => json_encode($event), ]) === 1; // 3. Queue and answer — the timeout is 10 seconds. if ($isNew) { SalafEventJob::dispatch($eventId); } return response()->noContent(200); } public function process(array $event): void { $object = $event['data']['object']; // 4. Upsert: order.updated can arrive before order.created. match ($event['type']) { 'order.created', 'order.updated', 'order.paid' => $this->upsertOrder($object), 'inventory.updated' => $this->setLevels($object['variant_id'], $object['levels']), default => null, // unknown types are normal }; } ``` The rest of this page is why each of those four lines is where it is. ## 1. Verify first Anyone can `POST` JSON at a public URL. Until the signature checks out, the body is a stranger's input — do not parse it, log it, or let it near a database. Three rules, all of which the samples on [Verifying signatures](https://www.salafems.com/developers/webhook-signatures.md) implement: - Hash the **raw bytes**, never a re-serialised object. - Compare in **constant time** (`timingSafeEqual`, `hmac.compare_digest`, `hash_equals`). - Reject a timestamp more than **five minutes** from your clock. Answer `400` when verification fails. We will retry, and every one of those retries lands in the merchant's delivery log where a human can see something is wrong — which is better than silently discarding requests that might have been ours. ## 2. De-duplicate on the event id **Duplicates are routine, not exceptional.** A timeout on our side is indistinguishable from a slow success on yours: you may have processed the event perfectly and lost the acknowledgement. So we retry, and you receive it again. The defence is one unique column: ```sql CREATE TABLE salaf_events ( id TEXT PRIMARY KEY, -- the evt_… id, verbatim type TEXT NOT NULL, payload JSONB NOT NULL, received_at TIMESTAMPTZ NOT NULL DEFAULT now(), handled_at TIMESTAMPTZ ); ``` Insert with `ON CONFLICT DO NOTHING`. If the insert did nothing, you have seen this event: answer `200` and stop. If it inserted, you own it. > **Warning — De-duplicate on the EVENT id, not the delivery id** > > `X-Salaf-Event-Id` (`evt_…`) identifies the fact and is **identical on every > retry**. `X-Salaf-Delivery` (`whd_…`) identifies the attempt and is > **different every time**. > > A receiver keyed on the delivery id de-duplicates nothing — every retry looks > new, and every retry runs again. Log the delivery id; key on the event id. > **Note — Storing the raw event pays for itself** > > Keeping the payload as you received it turns "why is this order wrong?" into > a query rather than an investigation, and lets you re-run a handler after > fixing a bug without asking the merchant to re-send anything. Our own copy is > kept for 30 days; yours can be kept for as long as you like. ## 3. Answer fast, work later The delivery times out after **10 seconds**. That is generous for "I have it" and deliberately hostile to "let me do the work first". Answer as soon as the event is durably recorded, then process it from a queue. A handler that talks to a payment provider, rebuilds a cache or sends an email inline will eventually exceed the timeout — and a timeout is not just slow, it produces a retry, so the slow work runs twice. If you have no queue, the minimum viable version is: insert the event row, answer `200`, and let a worker poll the table for `handled_at IS NULL`. That is a queue, and it is ten lines. ## 4. Upsert, in any order **Ordering is not guaranteed.** Deliveries run in parallel and retries reshuffle everything, so `order.updated` can arrive before `order.created`. This is much less painful than it sounds, because every payload carries the **whole object** as of the transaction that produced it. So: - **Write the object wholesale.** `INSERT … ON CONFLICT … DO UPDATE`, or your ORM's upsert. Never patch field-by-field from an event. - **Handle the create you never saw.** An update for an unknown id should create the record from `data.object`, not drop the event. - **Guard with a timestamp if you keep your own state machine.** Ignore a payload whose `data.object.updated_at` is older than what you already stored, and out-of-order delivery stops mattering entirely. - **`GET` the resource when in doubt.** The REST API is authoritative at the moment you call it. ```js // Last-write-wins, safe against out-of-order delivery. await db.query( `INSERT INTO orders (id, number, status, payload, updated_at) VALUES ($1, $2, $3, $4, $5) ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, payload = EXCLUDED.payload, updated_at = EXCLUDED.updated_at WHERE orders.updated_at < EXCLUDED.updated_at`, [order.id, order.number, order.status, order, order.updated_at], ); ``` ## Ignore what you do not know The catalog only ever grows, and a merchant subscribed to `*` receives new event types the day they ship. A handler that throws on an unrecognised `type` turns our additive change into your outage. `default: return`. Log it at debug level if you like. The same applies to fields: never validate a payload against a closed schema that rejects unknown keys. ## Failure modes worth handling | Situation | What to do | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Signature fails | `400`, log, alert if it is more than a stray. Check the [test vectors](https://www.salafems.com/developers/webhook-signatures.md#verify-your-implementation) before blaming the network. | | Your database is down | Return `5xx` **without** answering `2xx`. We retry on the ladder — this is exactly what it is for. | | A handler bug corrupts a record | Fix, then replay from your own stored events. No need to involve the merchant. | | You were down for days | Deliveries go `dead` after \~45 hours, and events are purged after 30 days. Reconcile with [`updated_after`](https://www.salafems.com/developers/pagination.md#syncing-changes-with-updated_after) rather than asking for a re-send. | | A merchant reports missing data | Ask for the `X-Salaf-Delivery` id from their delivery log; it pins the exact attempt and the exact payload we sent. | > **Danger — Never answer 2xx for an event you failed to record** > > A `200` means "I have this, do not send it again". If your insert failed and > you answer `200` anyway, the event is gone — we will not retry it, and no > amount of looking at our delivery log will show anything wrong. > > Answer `5xx` when you could not record it. That is what the retry ladder is > for, and it costs you nothing. ## Ready to go live Work through the [pre-launch checklist](https://www.salafems.com/developers/webhook-testing.md#a-checklist-before-you-go-live), then send a [`salaf.ping`](https://www.salafems.com/developers/webhook-testing.md) and watch it arrive.