It is 10:41 on a Tuesday in Nairobi.
A subscriber on a small ISP pays KES 1,000 through M-Pesa on a feature phone. Somewhere between that phone and the MikroTik router in a roadside cabinet, seven things have to happen, in order, exactly once.
If any of them happens twice, someone gets free internet.
If any of them happens zero times, a paying customer sits offline and phones the ISP owner, who phones me.
I have spent the last six months as the lead engineer on FyberPay, a multi-tenant billing and network platform for ISPs. Eight payment rails (M-Pesa STK Push, M-Pesa B2B, reversals, Paystack, KopoKopo, Tuma, Bill Manager, bank paybills). 876 HTTP endpoints. 263 hand-written SQL migrations. About 8,200 automated tests.
This is the part of that system I would rebuild the same way tomorrow: three layers of idempotency, and the circuit breakers around them.
The path a payment takes
Before the architecture, the physical route. Read it top to bottom.
phone ──▶ M-Pesa ──▶ webhook ──▶ FyberPay API ──▶ Postgres ledger
│
outbox worker (async)
│
┌───────────────────────────┼─────────────────────────┐
▼ ▼ ▼
extend subscription SMS / email RADIUS CoA (UDP)
│
▼
MikroTik router: onlineThree places in that diagram break in production, and they break in different ways:
- M-Pesa retries. Daraja will fire the same callback again if it does not get a fast 200. Twice is normal. Three times happens.
- The ledger. If the same receipt lands twice, the subscriber's balance is credited twice. That is the free-internet bug.
- The gap after the write. The process can die after the payment row commits but before the RADIUS packet goes out. Money in, no internet. That is the angry-phone-call bug.
Each layer below exists for one of those three.
Layer 1: the interceptor, and why the textbook version leaks
Every idempotency tutorial says the same thing: take the client's Idempotency-Key header, look it up in Redis, replay the cached response if you find one.
Here is what that looks like when two tenants on the same platform happen to send the same key.
The caller chooses the entire cache key. A collision, a reused UUID library seed, or a leaked key means one ISP can read another ISP's cached response body. The system is idempotent. It is not safe.
FyberPay's interceptor refuses to trust the header alone. It builds the key from the server's view of the request first, and only then appends the client's key at the end:
// idempotency.interceptor.ts (NestJS)
const userId = (request.user as { id?: string } | undefined)?.id;
const orgId = (request.org as { id?: string } | null | undefined)?.id ?? 'root';
const principal = userId ?? `anon:${request.ip ?? 'noip'}`;
const path = String(request.originalUrl ?? request.url ?? '').split('?')[0];
const bodyHash = createHash('sha256')
.update(JSON.stringify(request.body ?? {}))
.digest('hex')
.slice(0, 32);
const redisKey =
`${KEY_PREFIX}${principal}:${orgId}:${method}:${path}:${bodyHash}:${idempotencyKey}`;
// Claim the key atomically. If another request already holds it, we replay.
const claimed = await this.redis.set(redisKey, 'processing', 'EX', IDEMPOTENCY_TTL, 'NX');Read the key left to right and you can see the guarantee:
idem : u_8812 : org_kisumu-fiber : POST : /payments/stk : 9e1c...f2a0 : 7f3c-...-91ab
▲ ▲ ▲ ▲ ▲ ▲
who which tenant verb route body hash client's keySame header from two tenants produces two different Redis slots. There is no configuration in which Tenant B can be handed Tenant A's payload, because Tenant B's principal and org are baked into the address before the lookup happens.
SET ... NX matters too. The claim is atomic. Two concurrent retries of the same request race for one slot; exactly one wins and runs the handler, the other waits and replays.
Cost of this decision: it is stricter than the spec. A client that changes one byte of the body gets a fresh execution instead of a replay. In a payments system that is the behaviour you want.
Layer 2: the database refuses duplicates
Redis is fast and it forgets. TTLs expire. Providers sometimes retry through a path that never touches the HTTP interceptor (more on that in the timeline below).
So the second layer does not live in application code at all. It lives in PostgreSQL, in a hand-written migration:
-- migrations/0187_payment_receipt_unique.sql
ALTER TABLE payments
ADD CONSTRAINT payments_receipt_unique UNIQUE (gateway, receipt_number);
CREATE INDEX CONCURRENTLY IF NOT EXISTS payments_receipt_lookup
ON payments (gateway, receipt_number);An M-Pesa receipt number (RJK4T7XXXX) is globally unique on Safaricom's side. Storing it under a UNIQUE constraint means a replayed webhook that somehow slips past Redis meets the database and stops:
INSERT INTO payments (gateway, receipt_number, amount, subscriber_id, ...)
VALUES ('mpesa', 'RJK4T7XXXX', 1000, 4471, ...);
ERROR: duplicate key value violates unique constraint "payments_receipt_unique"
DETAIL: Key (gateway, receipt_number)=(mpesa, RJK4T7XXXX) already exists.The transaction rolls back. Nothing downstream of the insert ever sees the second copy.
One design choice hides inside this: raw SQL owns the schema; Prisma is only a read model. That rule was written after a prisma db push silently dropped a PostGIS column on staging and bricked user inserts. Every migration since has been hand-written and idempotent, 263 of them at the time of writing. The unique constraint above is the kind of thing you never want an ORM to "helpfully" drop.
What the three layers look like together
Redis is speed. Postgres is truth. The third layer is durability.
Layer 3: nothing important happens inside the request
Here is the failure that layers 1 and 2 cannot catch.
The webhook arrives. The interceptor claims the key. The payment row inserts cleanly. And then, before the subscription end date is extended or the RADIUS packet leaves the box, the container gets recreated by a deploy, or the node runs out of memory, or the network to the router drops.
The ledger says paid. The router says no. The subscriber is offline with a receipt in hand.
The fix is to never do side effects inside the request at all:
// payments.service.ts (simplified)
await this.db.transaction(async (tx) => {
const payment = await tx.payments.insert({ gateway, receiptNumber, amount, subscriberId });
await tx.outboxEvents.insert({
type: 'PAYMENT_CONFIRMED',
aggregateId: payment.id,
payload: { subscriberId, amount, tenantId },
status: 'pending',
});
});
// COMMIT. Both rows exist, or neither does.A worker pool claims pending events with a query that lets many workers run without stepping on each other:
UPDATE outbox_events
SET status = 'claimed', claimed_at = now(), attempts = attempts + 1
WHERE id = (
SELECT id FROM outbox_events
WHERE status = 'pending' AND next_attempt_at <= now()
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING *;SKIP LOCKED is the whole trick. Ten workers hit the table at once; each takes a different row; none of them block. If a worker dies mid-way, its row's lock is released, the row is still pending or claimed with a stale claimed_at, and a reaper puts it back in the queue.
The side effects that hang off that event in FyberPay:
- extend the subscription end date (credit-aware, so a partial payment does the right thing)
- send the SMS or email receipt (nine SMS gateways behind one interface, that is its own article)
- send a RADIUS Change-of-Authorization packet over raw UDP to the NAS, with an MD5 authenticator and retries, so the router lets the subscriber back on without waiting for the next session
Every one of those is at-least-once. That is only safe because layers 1 and 2 already guarantee the ledger moved exactly once. The outbox does not need to be exactly-once; it needs to be eventually, and idempotently on the receiving side. A second CoA packet for the same session is harmless. A second ledger credit is not.
A real duplicate, minute by minute
This is a reconstruction from logs of a normal Tuesday, not an incident. It is what "resilience" looks like when it is working:
10:41:02.114 webhook #1 RJK4T7 redis MISS -> claim -> INSERT ok -> outbox row -> 200
10:41:02.980 webhook #2 RJK4T7 redis HIT -> replay 200 (handler never ran)
10:41:31.400 webhook #3 RJK4T7 redis HIT -> replay 200 (Daraja gave up waiting)
11:15:07.000 hourly Daraja pull RJK4T7 -> INSERT fails UNIQUE -> rollback, loggedFour arrivals through two different code paths. One ledger row. One SMS. One CoA packet.
Notice the last line. FyberPay also pulls transactions from Daraja on a schedule, as a reconciliation net for webhooks that never arrive. That pull path never goes through the HTTP interceptor, so Redis cannot help it. Layer 2 is what makes reconciliation safe to run aggressively.
Circuit breakers that know the difference between a decline and an outage
Every gateway client in FyberPay sits behind a circuit breaker. The naive version counts every error. The naive version is wrong for mobile money.
Picture lunchtime. Fifty subscribers get the STK prompt, and a dozen of them fat-finger their PIN. Every one of those comes back from Safaricom as an error. If the breaker counts them, it opens, and now the other thirty-eight subscribers who typed their PIN correctly get "payment temporarily unavailable" for the next sixty seconds. You have turned customer behaviour into a self-inflicted outage.
So the client classifies before it counts:
// gateway-client.ts
type Outcome = 'provider_failure' | 'customer_decline' | 'success';
function classify(err: GatewayError): Outcome {
if (err.kind === 'timeout' || err.kind === 'network') return 'provider_failure';
if (err.httpStatus && err.httpStatus >= 500) return 'provider_failure';
if (err.kind === 'malformed_callback') return 'provider_failure';
// M-Pesa result codes for things the customer did
if (['1', '1032', '1037', '2001', '1025'].includes(err.resultCode)) {
return 'customer_decline'; // insufficient funds, cancelled, timeout at prompt, wrong PIN, limit
}
return 'provider_failure';
}
const outcome = classify(err);
if (outcome === 'provider_failure') breaker.recordFailure();
if (outcome === 'customer_decline') breaker.recordSuccess(); // the gateway did its jobAnd the breaker instance is keyed, not global:
const breaker = this.breakers.get(`${tenantId}:${gatewayName}`);That second line is the multi-tenant lesson. Early on there was one breaker per gateway. One ISP tenant with a misconfigured callback URL generated a stream of malformed responses, the shared breaker opened, and every other tenant on the platform lost M-Pesa for a minute at a time until someone noticed. Per-tenant breakers mean one customer's broken router cannot trip the circuit for everyone else.
What this cost, honestly
None of this is free, and the honest version of this article says so.
Four decisions, and what each one actually cost.
Server-namespaced idempotency keys, in place of the client's key as the whole cache address. Cost: stricter than the spec, so a changed body never replays. Bought: cross-tenant response leakage is structurally impossible.
A unique receipt constraint in hand-written SQL, in place of an ORM-managed schema. Cost: 263 idempotent migrations written by hand. Bought: reconciliation can be run aggressively, and no ORM can ever drop the constraint.
A transactional outbox with SKIP LOCKED, in place of side effects inside the request. Cost: workers, a reaper, and a status column to reason about. Bought: a crash between the commit and the SMS loses nothing.
Classifying breakers, one per tenant, in place of one counter per gateway. Cost: a hand-maintained result-code table for each gateway. Bought: wrong PINs do not open the breaker, and one tenant cannot trip the circuit for everyone else.
The pattern behind all four decisions: decide where the truth lives, and make everything else replayable. Redis can lie (expire). Workers can die. Webhooks can arrive four times. The ledger row with its unique constraint is the one thing that must be right, and everything else is built to converge on it.
Where to go from here
If you are wiring M-Pesa, Paystack or a bank paybill into something that has to be right, start with the unique receipt constraint. It is one migration and it is the layer that saves you when the clever layers fail.
The full FyberPay case study, with more of the RADIUS and reconciliation internals, is at kiragu.alkenacode.dev.
I take on fixed-scope payment engine builds, M-Pesa webhook hardening, and production architecture audits. Fastest reply is WhatsApp: +254 714 313 598. Email kiragu@alkenacode.dev, or pick a fixed-price engagement on Contra.