Transactional Inbox: The Handler That Ran Twice

Billing charged the card. Then the process died before it committed the Kafka offset. The partition was assigned to another instance. The same OrderPlaced record arrived again. The handler did not know it had already run. It charged again.

The outbox did its job. The event existed if and only if the order existed. The broker delivered it at least once, because that is what a crash-after-work-before-ack looks like. The producer pattern does not make the consumer run once. That is a different table, written in the same transaction as the side effect. That table is the inbox.

This is the consumer half of the pair. The outbox post already tells you to record processed ids. This post is what that record actually is, where it has to live, and which keys are lies.

At-least-once is the contract

A handler that does work and then acknowledges has a window:

1. receive message
2. apply side effect          ← credit posted, row updated, email sent
3. acknowledge                ← commit offset, DeleteMessage, complete

Die between 2 and 3 and the broker still owns the message. A restart, a rebalance, or a visibility timeout will give it to someone again. Kafka can commit a produce and a consumer offset together — exactly-once inside Kafka. It does not wrap a Postgres UPDATE or an HTTP call. SQS is the same window: process, then DeleteMessage; fail to delete and another worker receives the same MessageId. FIFO producer dedup does not remember that your handler already posted the credit.

The consumer is at-least-once. Design for a duplicate. The inbox is how you make the duplicate a no-op.

The skip has to be the same commit as the work

A Redis SETNX of the message id, then the SQL write, then the ack, is three systems. Crash after the write and before Redis, and the retry does the write again. Crash after Redis and before the write, and you skip work that never happened. Eviction, a failover that lost the key, a TTL shorter than your replay window — any of those forgets the skip. The skip is only real if it is stored where the side effect is stored, in one transaction.

CREATE TABLE inbox (
  consumer_name text        NOT NULL,
  message_id    uuid        NOT NULL,
  processed_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_name, message_id)
);

consumer_name is there because OrderPlaced is not one handler. Billing and inventory both consume it. Each needs its own skip. A single global set of ids would let inventory treat billing's success as its own.

The handler inserts first. Unique violation means this consumer already applied this id. Ack and return. Otherwise do the work in the same transaction and commit, then ack.

BEGIN;

INSERT INTO inbox (consumer_name, message_id)
VALUES ('billing', :message_id)
ON CONFLICT (consumer_name, message_id) DO NOTHING
RETURNING message_id;

-- no row returned: already processed. ROLLBACK. ack. return.

UPDATE accounts
   SET balance_cents = balance_cents + :amount_cents
 WHERE id = :account_id;

COMMIT;
-- then commit the Kafka offset / delete the SQS message

ON CONFLICT DO NOTHING is the race with a second delivery while the first is still in flight. One insert wins. The other gets no row, rolls back, acks. You do not need FOR UPDATE on the inbox for this shape. The unique key is the lock.

Ack after commit. Ack before commit and a crash loses the work: the inbox row is gone (rolled back) and the broker is done. You will not see that message again. That is a silent drop. The duplicate you get from commit-then-crash-before-ack is the cheaper failure. The inbox turns it into a skip.

What the message id actually is

The inbox key must name the logical event, not the transport attempt.

Producer event id, copied from the outbox. This is the one that works. The outbox row has an id. Put that id in the payload (or a header) and never change it when the relay publishes again. A crash in the relay republishes the same id. Both deliveries hit the same inbox row.

Kafka topic, partition, and offset. Fine for "I will never rewind this consumer group." A replay from an earlier offset reuses those numbers for the same records, which is what you want, until someone produces the event again onto a new offset — the outbox relay after a bug-fix, a restored topic, a new consumer group. New offset, old business event, second charge. Do not use the offset as the inbox key if the producer can emit the same event id twice.

A hash of the payload. Two legitimate events can have the same body: two one-dollar top-ups, two "mark shipped" commands with the same fields. You will skip the second. Hashing is not identity.

SQS MessageId. It is stable across receives of that message, which is enough for "this receipt came back." It is not enough if a producer retried the send and SQS accepted two messages. Standard queues do not producer-dedup. Put your own id in the body.

The idempotency key on a write API is the same idea with a different sender. The client of an HTTP POST chooses the key because the client is the one who retries. Here the producer of the event chooses the key because the producer is the one whose relay retries. Do not mint a new UUID in the consumer. The consumer is not allowed to invent identity for a message it might see twice.

Side effects the database cannot wrap

The inbox transaction covers what that database can roll back. A card capture, an email, a call to another service, a write to a different cluster — those commits are not in the BEGIN. The inbox row will not undo them.

For an HTTP side effect that itself supports an idempotency key, send the same message id as that key, every attempt. The first capture succeeds. The duplicate delivery sends the same key. The provider returns the original result and does not capture again. Your inbox still records that this consumer applied this id, so you do not keep hammering the provider on every replay. The inbox is the local skip. The provider key is the remote skip. You want both, because either commit can survive while the other does not.

If the remote call has no such key, a table in Postgres cannot make the pair atomic. You can make it recoverable: record "started," perform the call, mark "done." A crash after the call retries it — which is why the remote key still matters. Without one you are guessing. That is the class of problem two-phase commit pretends to solve across two systems, and the reason you do not run 2PC against a payment processor. "Exactly-once email" is not a consumer setting.

Cleanup, ordering, and what this is not

The table grows. Delete (or archive) rows older than the longest window you will rewind: the Kafka retention you actually use, the SQS redrive horizon, the outbox replay you keep. Delete inside that window and a replay looks new. The inbox is not an audit log. If you need the payload forever, that is a different store. This table is a set of ids.

Ordering is not included. The inbox says "I have seen this id," not "I have seen every id before it." Per-entity order still comes from a partition key on the broker, the same rule as the outbox post.

It is not a work lease. Claiming a job so another worker does not take it is SKIP LOCKED and a timeout. The inbox is the done set. Mixing "I am working" and "I finished" in one row is how a crashed worker leaves a poison id stuck until a human deletes it. When the work commits, insert the inbox row.

The pair

Draw the two tables next to each other and the job of each is obvious.

Side	Table	Written with	Failure it removes
Producer	outbox	The business row	Event never published
Consumer	inbox	The consumer's side effect	Event applied twice

Without the outbox you lose messages. Without the inbox you double-apply the ones you did not lose. Relays, CDC, and "we enabled Kafka transactions" do not fill the consumer side. A unique key on the business table sometimes does — an INSERT of order_id that already exists — and when the handler is exactly one insert, that unique key is the inbox. The moment the handler is an UPDATE, two side effects, or a row that is allowed to exist before this event, you need the extra id.

Put the producer event id in the payload. Insert it into inbox in the same BEGIN as the credit. Commit. Then ack. The second delivery finds the row, does nothing, and acks. That is the whole pattern. The handler in the first paragraph charged twice because step 2 was a side effect with no row that said it had happened. Add the row.