Handle webhooks idempotently
Verify, dedupe, upsert, and answer fast.
A correct webhook receiver is four steps in a fixed order. Get the order wrong and you have a receiver that works in testing and corrupts data in production.
1. verify the signature ─ before parsing, against the raw bytes
2. de-duplicate on event id ─ delivery is at-least-once
3. answer 2xx ─ within 10 seconds, before doing the work
4. upsert from data.object ─ off the request, in any orderThe whole receiver
app.post(
"/webhooks/salaf",
express.raw({ type: "application/json" }),
async (req, res) => {
// 1. Verify BEFORE parsing or trusting anything.
const ok = verifySalafSignature(
req.body,
req.get("X-Salaf-Signature"),
process.env.SALAF_WEBHOOK_SECRET,
);
if (!ok) {
return res.sendStatus(400); // we will not retry a 4xx into working
}
const event = JSON.parse(req.body.toString("utf8"));
const eventId = req.get("X-Salaf-Event-Id"); // === event.id
// 2. Dedupe. Delivery is at-least-once: the same event WILL arrive
// twice eventually. A unique index on the event id is the whole trick.
const inserted = await db.query(
`INSERT INTO salaf_events (id, type, payload)
VALUES ($1, $2, $3) ON CONFLICT (id) DO NOTHING`,
[eventId, event.type, event],
);
// 3. Answer FAST. Anything slower than a few hundred ms risks our 10s
// timeout, and a timeout means a retry you did not need.
res.sendStatus(200);
if (inserted.rowCount === 0) {
return; // already handled — nothing more to do
}
// 4. Do the work off the request. Order is NOT guaranteed, so write
// upserts: order.updated may land before order.created.
void queue.add("salaf-event", { eventId });
},
);
async function handleEvent(event) {
switch (event.type) {
case "order.created":
case "order.updated":
case "order.paid":
// data.object is the full snapshot as it was inside the transaction
// that produced it — upsert it, do not patch field by field.
return upsertOrder(event.data.object);
case "inventory.updated":
return setLevels(event.data.object.variant_id, event.data.object.levels);
default:
return; // unknown types are normal: the catalog only ever grows
}
}import json
from django.db import IntegrityError
from django.http import HttpResponse
@csrf_exempt
@require_POST
def salaf_webhook(request):
# 1. Verify before trusting the body.
if not verify_salaf_signature(
request.body,
request.headers.get("X-Salaf-Signature"),
settings.SALAF_WEBHOOK_SECRET,
):
return HttpResponse(status=400)
event = json.loads(request.body)
event_id = request.headers["X-Salaf-Event-Id"] # same as event["id"]
# 2. Dedupe on the event id — delivery is at-least-once.
try:
SalafEvent.objects.create(
id=event_id, type=event["type"], payload=event
)
except IntegrityError:
return HttpResponse(status=200) # already seen; still a success
# 3. Queue the work and answer immediately: our timeout is 10 seconds,
# and a slow 200 costs you a duplicate you did not need.
process_salaf_event.delay(event_id)
return HttpResponse(status=200)
def handle_event(event):
kind = event["type"]
obj = event["data"]["object"]
# 4. Upsert, never patch: events can arrive out of order, and every
# payload carries the complete object as of its own transaction.
if kind in ("order.created", "order.updated", "order.paid"):
upsert_order(obj)
elif kind == "inventory.updated":
set_levels(obj["variant_id"], obj["levels"])
# Unknown types are expected — the catalog only ever grows.<?php
public function handle(Request $request)
{
// 1. Verify first.
if (! verify_salaf_signature(
$request->getContent(),
$request->header('X-Salaf-Signature'),
config('services.salaf.webhook_secret'),
)) {
return response()->noContent(400);
}
$event = json_decode($request->getContent(), true);
$eventId = $request->header('X-Salaf-Event-Id'); // same as $event['id']
// 2. Dedupe on the event id (unique column). At-least-once delivery
// means duplicates are routine, not exceptional.
$isNew = SalafEvent::query()->insertOrIgnore([
'id' => $eventId,
'type' => $event['type'],
'payload' => json_encode($event),
]) === 1;
// 3. Queue and answer — the timeout is 10 seconds.
if ($isNew) {
SalafEventJob::dispatch($eventId);
}
return response()->noContent(200);
}
public function process(array $event): void
{
$object = $event['data']['object'];
// 4. Upsert: order.updated can arrive before order.created.
match ($event['type']) {
'order.created', 'order.updated', 'order.paid' => $this->upsertOrder($object),
'inventory.updated' => $this->setLevels($object['variant_id'], $object['levels']),
default => null, // unknown types are normal
};
}The rest of this page is why each of those four lines is where it is.
1. Verify first
Anyone can POST JSON at a public URL. Until the signature checks out, the
body is a stranger's input — do not parse it, log it, or let it near a
database.
Three rules, all of which the samples on Verifying signatures implement:
- Hash the raw bytes, never a re-serialised object.
- Compare in constant time (
timingSafeEqual,hmac.compare_digest,hash_equals). - Reject a timestamp more than five minutes from your clock.
Answer 400 when verification fails. We will retry, and every one of those
retries lands in the merchant's delivery log where a human can see something is
wrong — which is better than silently discarding requests that might have been
ours.
2. De-duplicate on the event id
Duplicates are routine, not exceptional. A timeout on our side is indistinguishable from a slow success on yours: you may have processed the event perfectly and lost the acknowledgement. So we retry, and you receive it again.
The defence is one unique column:
CREATE TABLE salaf_events (
id TEXT PRIMARY KEY, -- the evt_… id, verbatim
type TEXT NOT NULL,
payload JSONB NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
handled_at TIMESTAMPTZ
);Insert with ON CONFLICT DO NOTHING. If the insert did nothing, you have seen
this event: answer 200 and stop. If it inserted, you own it.
3. Answer fast, work later
The delivery times out after 10 seconds. That is generous for "I have it" and deliberately hostile to "let me do the work first".
Answer as soon as the event is durably recorded, then process it from a queue. A handler that talks to a payment provider, rebuilds a cache or sends an email inline will eventually exceed the timeout — and a timeout is not just slow, it produces a retry, so the slow work runs twice.
If you have no queue, the minimum viable version is: insert the event row,
answer 200, and let a worker poll the table for handled_at IS NULL. That is
a queue, and it is ten lines.
4. Upsert, in any order
Ordering is not guaranteed. Deliveries run in parallel and retries reshuffle
everything, so order.updated can arrive before order.created.
This is much less painful than it sounds, because every payload carries the whole object as of the transaction that produced it. So:
- Write the object wholesale.
INSERT … ON CONFLICT … DO UPDATE, or your ORM's upsert. Never patch field-by-field from an event. - Handle the create you never saw. An update for an unknown id should
create the record from
data.object, not drop the event. - Guard with a timestamp if you keep your own state machine. Ignore a
payload whose
data.object.updated_atis older than what you already stored, and out-of-order delivery stops mattering entirely. GETthe resource when in doubt. The REST API is authoritative at the moment you call it.
// Last-write-wins, safe against out-of-order delivery.
await db.query(
`INSERT INTO orders (id, number, status, payload, updated_at)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (id) DO UPDATE
SET status = EXCLUDED.status,
payload = EXCLUDED.payload,
updated_at = EXCLUDED.updated_at
WHERE orders.updated_at < EXCLUDED.updated_at`,
[order.id, order.number, order.status, order, order.updated_at],
);Ignore what you do not know
The catalog only ever grows, and a merchant subscribed to * receives new
event types the day they ship. A handler that throws on an unrecognised type
turns our additive change into your outage.
default: return. Log it at debug level if you like.
The same applies to fields: never validate a payload against a closed schema that rejects unknown keys.
Failure modes worth handling
| Situation | What to do |
|---|---|
| Signature fails | 400, log, alert if it is more than a stray. Check the test vectors before blaming the network. |
| Your database is down | Return 5xx without answering 2xx. We retry on the ladder — this is exactly what it is for. |
| A handler bug corrupts a record | Fix, then replay from your own stored events. No need to involve the merchant. |
| You were down for days | Deliveries go dead after ~45 hours, and events are purged after 30 days. Reconcile with updated_after rather than asking for a re-send. |
| A merchant reports missing data | Ask for the X-Salaf-Delivery id from their delivery log; it pins the exact attempt and the exact payload we sent. |
Ready to go live
Work through the
pre-launch checklist,
then send a salaf.ping and watch it arrive.