Designing a Payment System: Double-Entry Ledgers and Idempotent Money

Payment systems run on the same infrastructure as everything else — queues, databases, retries — but with one property that changes the engineering culture entirely: errors do not average out, they accumulate as missing money. A feed that drops 0.01% of posts is fine. A ledger that drops 0.01% of transactions is a company-ending audit. So payment architecture is less about scale and more about a small set of invariants that must hold under every crash, retry, and race.

Three of them do most of the work: money is never created or destroyed (only moved), every state change is attributable and immutable, and every external effect happens exactly once from the customer's point of view.

The balance column is a lie

The intuitive schema — accounts(id, balance) and UPDATE accounts SET balance = balance + 100 — fails all three invariants at once. A crash between debiting one row and crediting another silently destroys money. An overwritten balance has no history: when a customer disputes a charge, "the number is 420 now" is not an answer. And concurrent updates without care lose writes.

The 700-year-old fix is double-entry bookkeeping, which maps directly onto an append-only schema:

CREATE TABLE ledger_entries (
  id             BIGINT PRIMARY KEY,
  transaction_id BIGINT NOT NULL,      -- groups the legs
  account_id     BIGINT NOT NULL,
  direction      TEXT NOT NULL,        -- debit | credit
  amount_minor   BIGINT NOT NULL CHECK (amount_minor > 0),
  currency       TEXT NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- invariant: per transaction_id, sum(debits) == sum(credits)

Every money movement is a transaction with two or more legs that sum to zero: charge a customer $10 with a $0.30 fee and you write three legs — customer minus 1030, revenue plus 1000, fees plus 30 (in minor units; never floats — binary floating point cannot represent 0.1, and rounding errors in money are incidents, not noise). The rows are immutable: no UPDATE, no DELETE. Corrections are new reversal entries, which is exactly what refunds are. A balance is now a derived fact — SUM(credits) - SUM(debits) — with a cached running balance maintained transactionally as an optimization, verifiable against the entries at any time.

What this buys: money conservation enforced by a sum-to-zero check, a complete audit trail for free, and no lost-update races on a balance column. One real scaling wrinkle — hot accounts: every sale touches the revenue account, serializing on one row. Standard fix is sub-accounts (shard the revenue account into N, sum for reporting), the hot-partition medicine applied to bookkeeping.

Idempotency: the network will make you charge twice

The nightmare sequence: client calls POST /charge, the charge succeeds, the response times out, the client retries, the customer pays twice. Retries are mandatory (networks fail) and dangerous (money moves), so every money-moving API takes an idempotency key — a client-generated unique value per intent:

POST /v1/charges
Idempotency-Key: order-8812-attempt

server: INSERT INTO idempotency_keys(key, ...) -- unique constraint
        ON CONFLICT → return the stored response of the original attempt
        else        → process, store response atomically with the ledger write

The unique constraint makes the race safe: two concurrent retries collide in the database and exactly one processes. Stripe's API works precisely this way, and the same discipline applies inside your system — the worker consuming payment.captured events will see duplicates (at-least-once delivery guarantees it) and must dedupe on event ID before writing ledger entries.

State machines carry the other half of correctness. A payment is not a boolean; it walks created → authorized → captured → settled, branching to failed, refunded, disputed. Model transitions as compare-and-swap — UPDATE payments SET state='captured' WHERE id=? AND state='authorized' — so an out-of-order or duplicate webhook from your payment provider (they are also at-least-once) becomes a no-op instead of a double-capture.

Reconciliation: trust, then verify with a file

Your ledger says what you think happened. The card network, the bank, and the payment processor each have their own opinion, delivered as settlement files — and the amounts will disagree: fees, currency conversion, chargebacks, transactions that settled on their side but errored on yours. Reconciliation is the batch job that joins your ledger against their records, line by line, and files every mismatch into a workflow: match, missing-ours, missing-theirs, amount-differs.

This is not optional hygiene; it is the payment system's immune system. Teams that treat reconciliation as an afterthought discover money problems months late, at audit time, when the trail is cold. Design it in from day one: every external transaction carries your ID into their system (so the join has a key), and every day closes with a zero-unexplained-differences report. The transactional outbox pattern matters here too — ledger writes and the events that announce them must not diverge, or your own downstream systems become another reconciliation problem.

Takeaways

Payment systems are invariant-preservation machines. Append-only double-entry ledger entries in minor units, transactions whose legs sum to zero, balances as derived data. Idempotency keys on every external mutation and CAS-guarded state machines on every internal one, because retries and duplicate webhooks are the steady state, not the exception. Reconciliation against every external party, daily, with unexplained differences treated as incidents. None of this is glamorous — and that is the point: in payments, boring and provably consistent beats clever every single time.