Independent reconciliation for Stripe webhooks

A ledger that admits
what it never received.

Every payment recorded once, and only once. What matters is the payment that was never recorded at all — and the fact that nothing else reported it. Inventing a row to make the books look complete is the one thing this was built to prevent.

Read the incident See how it works
Recorded once The database decides what is a duplicate, not the code
Independent check Asks Stripe directly what it sent, then compares
One real incident A genuine gap from this system's own first hour, not staged
Nothing installed Reconciliation runs on a schedule, outside the write path

The blind spot

A green dashboard can be quietly lying

Most billing monitoring can only report what arrived. The whole failure mode here is a payment that never arrived at all — which every ordinary log, alert and dashboard is structurally unable to see.

A rejected webhook looks like nothing happened

A signature check that fails returns an error and stops there. The payment already happened — only the announcement of it was lost, and nothing retries a message that was refused as invalid.

No error, no alert, nothing to click on

Every log and every dashboard can be genuinely healthy while a customer has paid for something the records say they never bought. Green is not the same claim as correct.

No alarm can catch what never arrived

The only way to find a payment that was never recorded is to ask the sender what it actually sent and compare. Almost nothing does this, because there is no error to prompt it.

How it works

Four steps, only one of them inside the write path

Recording and reconciling are kept apart on purpose. The write path never asks Stripe anything; the reconciliation job never writes to the ledger.

  1. 01

    Record once

    A verified webhook is written to the ledger. The database's own key decides what counts as a duplicate — the code never has to guess.

    AWS Lambda DynamoDB
  2. 02

    Refuse, don't retry forever

    A webhook that fails its signature check is rejected outright. That is correct behaviour for a forged request — and the exact reason a genuine but misconfigured one can go missing quietly.

    Stripe Webhooks
  3. 03

    Ask the source directly

    On a schedule, a job asks Stripe what it actually sent in a window — the only way to find something that never triggered an error here.

    src/reconcile.py
  4. 04

    Report the gap, never invent it away

    Anything Stripe has that the ledger doesn't is reported as missing. It is never written in after the fact to make the books look complete.

    pytest + moto

The incident

What happened on 29 August, one step at a time

A real failure from this system's own first hour. Every line of output below is verbatim.

Live counters

Fetched from the real Lambda on load. A failure here must not look like a healthy empty state, so it is reported.

Loading live counters…

What you get

Four things, in plain language

Written to be read by whoever is deciding whether the books can be trusted, not only by whoever built the pipeline.

Immutable ledger

Every verified payment recorded once, deduplicated by the database's own key rather than application logic that could drift.

Answers: what did we actually receive?

Independent reconciliation

A scheduled check that asks Stripe directly, rather than trusting that silence means everything arrived.

Answers: does our record match theirs?

An honest gap

A missing payment is reported by name, not silently written in to make the total look complete.

Answers: what did we miss, specifically?

Live counters

Recorded and duplicate-suppressed counts, fetched live rather than quoted from a report that could be stale.

Answers: what is happening right now?

Find the payment your dashboard can't see

This ran against Stripe in test mode. The reconciliation works exactly the same way in production — the difference is whose money it is.

Ask about it See it on the live site