Skip to content

Idempotent consumers and the exactly-once myth

A consumer separates delivery identity from durable business identity.

Predict first

Before you read: predict

Job data: job-77, effect ledger-77, 500 cents; inbox.message_id is the primary key and effects has no unique key. The shipped handler ran once, then the pod died after COMMIT and before acknowledgement; a fresh worker received the same job.

What do the inbox row count, effect row count and effect sum show?

Code for this question
const job = { id: 'job-77', effectId: 'ledger-77', cents: 500 };const inboxId = job.id;const insertInbox = (id) => tx.query(  'INSERT INTO inbox(message_id) VALUES($1) ON CONFLICT DO NOTHING RETURNING message_id', [id]).then((r) => r.rowCount === 1);const insertEffect = (id, cents) => tx.query(  'INSERT INTO effects(effect_id, amount) VALUES($1,$2)', [id, cents]);await tx.begin();const claim = await insertInbox(inboxId);if (!claim) { await tx.rollback(); return; }await insertEffect(job.effectId, job.cents);await tx.commit();
Pick one

The 30-second version

A broker can redeliver work after a handler has changed durable state. A consumer gives each incoming message a durable identity, uses the database to arbitrate repeats, and gives each business effect an identity that matches the operation. Some state assignments can safely run again; additive effects need their own repeat-safe boundary.

A consumer separates delivery identity from durable business identity.

How it works

Step 1 of 10

A consumer can receive one logical message more than once after a restart, crash, or uncertain acknowledgement.

Analogy: Organ-pipe tuning workcards

A tuner opens a workcard for one pipe repair and records the repaired pipe on the same card. A second copy of that card is checked before another repair is logged. This illustrates identity and an effect; workshop cards are not transactions, and an outside delivery remains separate.

Tuning workcard Consumer
workcard number stable message identity
opened workcard inbox claim
pipe repair entry business effect
copied workcard duplicate path

Examples, run for real

4.1 Inbox conflict is a consumer decision

The fixture creates an inbox(message_id primary key) table and obtains the claim with INSERT ... ON CONFLICT DO NOTHING RETURNING. It executes a new-message path and a duplicate path against a fresh private PostgreSQL database. The database result, rather than elapsed time, decides whether a claim was new.

BEGIN;
INSERT INTO inbox(message_id) VALUES ($1) ON CONFLICT DO NOTHING RETURNING message_id;
-- only a returned row may continue with the consumer effect
COMMIT;

PostgreSQL documents ON CONFLICT DO NOTHING as an alternative action for a unique conflict [S1].

4.1 inbox claims: notice-amber=fresh, notice-birch=fresh, notice-amber=conflict

4.2 Preserve the identity of a state snapshot

A reconciliation arrival carries a snapshot identity and an absolute value. The fixture compares a repeat-safe state assignment with designs that confuse an arrival, worker, or account with that snapshot. This is a consumer identity choice: a new snapshot for the same account must remain distinguishable from its earlier snapshot.

4.2 snapshots: ids=snapshot-elm-01,snapshot-elm-02 claims=fresh,fresh balance=930

4.3 Rollback keeps a consumer claim retryable

The fixture begins a transaction, forces an in-transaction failure, and verifies that a later worker can retry the complete operation. The check uses database rows and transaction state; it does not treat a broker acknowledgement as a database guarantee. External providers still need their own idempotency contract [S2].

Each model uses a fresh worker connection, so worker-local memory cannot decide the result. The fixture records rows, identifiers, and sums only after each transaction finishes. It does not infer correctness from a timely acknowledgement, a process restart, or a sleep. This keeps the teaching evidence on durable consumer state rather than scheduling behavior across retries, restarts, and competing workers consistently.

4.3 rollback: fresh-worker-before inbox=0 effects=0; retry=applied; after inbox=1 effects=1

4.4 Effect identity follows the operation

One source message may represent a refund already described by another source message, while another message represents a different refund. The fixture stores source-message claims and business-effect identities separately, then checks both cardinality and the retained effect IDs. Kafka delivery semantics do not turn an external database write into exactly-once work [S3].

4.4 effect identity: source-maple-a=fresh/new, source-maple-b=fresh/already-recorded; inbox=2 effects=1

Try it

Answer first, then check. Nothing is saved.

Question 1 of 4

The worker runs this shipped handler for: result-81, 500; its replay result-81, 500 on another worker; then result-82, 300. The three deliveries use workers A, B, A and share course math-1. Each item starts with empty inbox and effect tables; insertInbox returns whether its unique insert created a row.

What durable tuple follows?

Code for this question
const inboxId = message.id;const effectId = message.id;if (await insertInbox(inboxId)) {  await db.query('INSERT INTO effects(effect_id, amount) VALUES($1,$2)', [effectId, message.cents]);}
Pick one

Question 2 of 4

Snapshots arrive as: reconcile-17, 700; its replay reconcile-17, 700; then reconcile-18, 900. They use workers A, B, A for account acct-9, with empty inbox and balance tables. insertInbox returns whether its unique insert created a row.

What durable tuple follows?

Code for this question
const inboxId = snapshot.id;const nextBalance = snapshot.cents;if (await insertInbox(inboxId)) {  INSERT INTO balances VALUES ('acct-9', nextBalance)  ON CONFLICT(account_id) DO UPDATE SET cents=excluded.cents;}
Pick one

Question 3 of 4

Three source messages passed the inbox check. Two describe one 400-cent refund; the third describes a different 300-cent refund. The first used worker A, the second worker B, and the third worker A; all three belong to one merchant. The observed database state is inbox=3, effects=1, sum=400.

Which faulty effect identity explains that trace?

Pick one

Question 4 of 4

The buggy handler uses an amount-only key. One sale is replayed through another worker; a second sale has the same amount. The three arrivals are 88 cents each, through workers A, B, A, with distinct delivery identifiers.

Which key repair preserves the business operation through that replay and same-amount arrival?

Code for this question
const effectId = "amount-" + message.cents;INSERT INTO effects(effect_id, amount) VALUES(effectId, message.cents)ON CONFLICT DO NOTHING;
Pick one

Go deeper

Pro tips
  1. Make the database arbitrate the consumer identity. Application-side “check then insert” races; a unique constraint gives every worker the same durable answer. When: adding an inbox or effect table. Verify: inspect the conflict branch and its database row count.
  2. Keep transport IDs for diagnosis, then name the business operation separately. A delivery attempt can be useful in logs while the durable effect needs a domain identity. When: a queue replay or backfill can arrive through another worker. Verify: compare the stored effect identity with the domain record it changes.
Sources
  • [S1] PostgreSQL, INSERT, fetched 2026-09-30; SHA-256 68FD1EE90EEBE03553C77D7E19337B4796995F70E5CB96CEF910DB48AD1D1E20.
  • [S2] Chris Richardson, Idempotent Consumer pattern, fetched 2026-09-30; SHA-256 989EE9AD8CCF53DF52D36CDFFF64EFFD770D6A3082A3E364E717BD6F2903AA8D. It describes recording processed message IDs and the same-database transaction approach used in steps 3–8.
  • [S3] Apache Kafka, Message Delivery Semantics, fetched 2026-09-30; final URL verified; SHA-256 A96A0FFE954F97531F810144A08D2037AC8A40E03DD91BA17F02B5E6F994C87F, saved in the pack run folder.
  • BullMQ, Idempotent jobs, fetched 2026-09-30. It supports the prerequisite link: a queued job may need repeat-safe work; this lesson owns the database boundary.

Related lessons

Checked against the tools and docs as of 2026-09-30.