SALAF EMSDevelopers
salafems.com

Handle webhooks idempotently

Verify, dedupe, upsert, and answer fast.

A correct webhook receiver is four steps in a fixed order. Get the order wrong and you have a receiver that works in testing and corrupts data in production.

1. verify the signature      ─ before parsing, against the raw bytes
2. de-duplicate on event id  ─ delivery is at-least-once
3. answer 2xx                ─ within 10 seconds, before doing the work
4. upsert from data.object   ─ off the request, in any order

The whole receiver

app.post(
  "/webhooks/salaf",
  express.raw({ type: "application/json" }),
  async (req, res) => {
    // 1. Verify BEFORE parsing or trusting anything.
    const ok = verifySalafSignature(
      req.body,
      req.get("X-Salaf-Signature"),
      process.env.SALAF_WEBHOOK_SECRET,
    );
    if (!ok) {
      return res.sendStatus(400); // we will not retry a 4xx into working
    }

    const event = JSON.parse(req.body.toString("utf8"));
    const eventId = req.get("X-Salaf-Event-Id"); // === event.id

    // 2. Dedupe. Delivery is at-least-once: the same event WILL arrive
    // twice eventually. A unique index on the event id is the whole trick.
    const inserted = await db.query(
      `INSERT INTO salaf_events (id, type, payload)
       VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING`,
      [eventId, event.type, event],
    );

    // 3. Answer FAST. Anything slower than a few hundred ms risks our 10s
    // timeout, and a timeout means a retry you did not need.
    res.sendStatus(200);

    if (inserted.rowCount === 0) {
      return; // already handled — nothing more to do
    }

    // 4. Do the work off the request. Order is NOT guaranteed, so write
    // upserts: order.updated may land before order.created.
    void queue.add("salaf-event", { eventId });
  },
);

async function handleEvent(event) {
  switch (event.type) {
    case "order.created":
    case "order.updated":
    case "order.paid":
      // data.object is the full snapshot as it was inside the transaction
      // that produced it — upsert it, do not patch field by field.
      return upsertOrder(event.data.object);
    case "inventory.updated":
      return setLevels(event.data.object.variant_id, event.data.object.levels);
    default:
      return; // unknown types are normal: the catalog only ever grows
  }
}

The rest of this page is why each of those four lines is where it is.

1. Verify first

Anyone can POST JSON at a public URL. Until the signature checks out, the body is a stranger's input — do not parse it, log it, or let it near a database.

Three rules, all of which the samples on Verifying signatures implement:

  • Hash the raw bytes, never a re-serialised object.
  • Compare in constant time (timingSafeEqual, hmac.compare_digest, hash_equals).
  • Reject a timestamp more than five minutes from your clock.

Answer 400 when verification fails. We will retry, and every one of those retries lands in the merchant's delivery log where a human can see something is wrong — which is better than silently discarding requests that might have been ours.

2. De-duplicate on the event id

Duplicates are routine, not exceptional. A timeout on our side is indistinguishable from a slow success on yours: you may have processed the event perfectly and lost the acknowledgement. So we retry, and you receive it again.

The defence is one unique column:

CREATE TABLE salaf_events (
  id          TEXT PRIMARY KEY,   -- the evt_… id, verbatim
  type        TEXT NOT NULL,
  payload     JSONB NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  handled_at  TIMESTAMPTZ
);

Insert with ON CONFLICT DO NOTHING. If the insert did nothing, you have seen this event: answer 200 and stop. If it inserted, you own it.

3. Answer fast, work later

The delivery times out after 10 seconds. That is generous for "I have it" and deliberately hostile to "let me do the work first".

Answer as soon as the event is durably recorded, then process it from a queue. A handler that talks to a payment provider, rebuilds a cache or sends an email inline will eventually exceed the timeout — and a timeout is not just slow, it produces a retry, so the slow work runs twice.

If you have no queue, the minimum viable version is: insert the event row, answer 200, and let a worker poll the table for handled_at IS NULL. That is a queue, and it is ten lines.

4. Upsert, in any order

Ordering is not guaranteed. Deliveries run in parallel and retries reshuffle everything, so order.updated can arrive before order.created.

This is much less painful than it sounds, because every payload carries the whole object as of the transaction that produced it. So:

  • Write the object wholesale. INSERT … ON CONFLICT … DO UPDATE, or your ORM's upsert. Never patch field-by-field from an event.
  • Handle the create you never saw. An update for an unknown id should create the record from data.object, not drop the event.
  • Guard with a timestamp if you keep your own state machine. Ignore a payload whose data.object.updated_at is older than what you already stored, and out-of-order delivery stops mattering entirely.
  • GET the resource when in doubt. The REST API is authoritative at the moment you call it.
// Last-write-wins, safe against out-of-order delivery.
await db.query(
  `INSERT INTO orders (id, number, status, payload, updated_at)
   VALUES ($1, $2, $3, $4, $5)
   ON CONFLICT (id) DO UPDATE
     SET status = EXCLUDED.status,
         payload = EXCLUDED.payload,
         updated_at = EXCLUDED.updated_at
     WHERE orders.updated_at < EXCLUDED.updated_at`,
  [order.id, order.number, order.status, order, order.updated_at],
);

Ignore what you do not know

The catalog only ever grows, and a merchant subscribed to * receives new event types the day they ship. A handler that throws on an unrecognised type turns our additive change into your outage.

default: return. Log it at debug level if you like.

The same applies to fields: never validate a payload against a closed schema that rejects unknown keys.

Failure modes worth handling

SituationWhat to do
Signature fails400, log, alert if it is more than a stray. Check the test vectors before blaming the network.
Your database is downReturn 5xx without answering 2xx. We retry on the ladder — this is exactly what it is for.
A handler bug corrupts a recordFix, then replay from your own stored events. No need to involve the merchant.
You were down for daysDeliveries go dead after ~45 hours, and events are purged after 30 days. Reconcile with updated_after rather than asking for a re-send.
A merchant reports missing dataAsk for the X-Salaf-Delivery id from their delivery log; it pins the exact attempt and the exact payload we sent.

Ready to go live

Work through the pre-launch checklist, then send a salaf.ping and watch it arrive.