Skip to content

Retries, poison messages and the dead-letter queue

Retry what time can fix, fail fast what it cannot, and keep every failure where someone is paged and can replay it.

Predict first

Before you read: predict

ExamHall's certificate worker renders each candidate's PDF from a template and uploads it to storage. The handler throws UnrecoverableError when a PDF will not parse or storage answers 404 for the template, and a plain Error when the upload times out. Jobs are added with attempts: 4 and exponential backoff; no other option is set. The malformed job's PDF never parses. The template job's upload times out on its first two runs; on its third, storage answers 404 for its template, as it would on every later run.

How many times does the handler run each job, and where does each end?

Pick one

The 30-second version

Sort every failure in the handler. Throw a plain error for a transient one: it spends an attempt and retries after a backoff. Throw UnrecoverableError for a permanent one: it fails at once. Cap attempts, so a poison job lands in the failed set with its payload. Page on arrivals; fix, then replay.

Retry what time can fix, fail fast what it cannot, and keep every failure where someone is paged and can replay it.

How it works

Step 1 of 13

Sort the failure where it happens. Transient: a timeout, a 5xx. Permanent: input that can never work (ds-retries-backoff-jitter step 1).

Analogy: An airport baggage sorting hall

Bags ride a loop of belts past scanners that read each tag and divert the bag to its flight.

In the baggage hall In BullMQ
a smudged tag the scanner misreads goes round the loop for another read transient: throw, retry after backoff (step 2)
a bag with no tag goes straight to the problem-bag room; another lap won't grow one UnrecoverableError (step 3)
the gate calls "hold, aircraft not ready" a 429: back off (step 4)
a bag that jams the diverter every lap goes to the problem room after three laps, with its tag record a poison job, the cap, the failed set (steps 6, 8)
the problem room's buzzer page on arrivals (step 10)
once the diverter is fixed, problem bags go back on the belt replay (steps 11–12)

In real terms: decide, for each error a handler can meet, whether time can fix it, and make sure every job that stops trying lands where it is seen and kept.

Breaks down when: a handler drags a jammed bag along the belt by hand; nothing in the hall stops it, just as nothing in BullMQ stops a retry loop inside a handler (step 7).

Contrasting case: a nightly report job that simply runs again tomorrow. Losing one run costs nothing, so a single attempt and a log line are enough.

Examples, run for real

4.1 ExamHall: the PDF that blocked every certificate

EXH-5040: after an exam, one candidate's malformed PDF reached the certificate worker (concurrency 1) first. The handler retried inside itself until it worked. MR !5340 sorts failures and caps attempts.

4.1 EXH-5040 before: retries inside the handler · after 1,000 tries (8.3 virtual hours of its own backoff): certificates issued 0 of 10,000 · malformed job active · waiting 10,000 · failed set 0
4.1 MR !5340 after: certificates issued 10,000 of 10,000 · malformed job 1 run, failed ("PDF will not parse: unexpected end of file") · pages 1 · failed set 1, holding {"candidate":"c-0417","exam":"e-2210"}

The in-handler loop kept the only slot (step 7). After the fix, the malformed job failed once and landed in the failed set with its payload; one page fired (steps 3, 8, 10).

4.2 Media streaming: what the failed set keeps

A corrupt upload fails with UnrecoverableError; a separate run shows where a retry without backoff goes.

4.2 corrupt upload in the failed set: data {"uploadId":"u-771","creator":"cr-19"} · failedReason "CORRUPT_INPUT: moov atom not found" · attemptsMade 1 of 5 · stacktrace entries 1
4.2 no backoff, one worker, attempts 3, a poison job first: run order poison u1 u2 u3 u4 poison poison · poison failed

The failed job keeps what a replay needs (step 8). With no backoff, a retried job goes to the back of the line, as a newly added job does, so others still run, but each retry costs a run (step 6).

Picture it

a worker places a repeatedly damaged parcel onto a dedicated review shelf while normal parcels continue through the workshop (the asset map's description).
An illustration to help it stick. The mechanism is in the sections above.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

A video platform's transcode worker runs this handler:

Jobs are added with attempts: 5 and exponential backoff. On-call is paged whenever a job lands in the failed set. A creator's corrupt upload never appears on the site, and nobody was paged.

Which is the bug?

Code for this question
new Worker('transcode', async (job) => {  try {    return await transcode(job.data.uploadId);  } catch (err) {    if (err.code === 'CORRUPT_INPUT') {      log.warn({ uploadId: job.data.uploadId }, 'corrupt upload');      return { skipped: true };    }    throw err;  }}, { connection });
Pick one

Question 2 of 6

A video platform runs four transcode worker processes, each with its own Worker; jobs have attempts: 5 and exponential backoff, and the handler throws UnrecoverableError for a corrupt upload. One line will page on-call. Goal: exactly one page for each job that lands in the failed set, corrupt uploads included, and no page for a run that will be retried.

Which line fills the slot?

Pick one

Question 3 of 6

A subtitle provider was down for two hours. 300 subtitle jobs (attempts: 4, exponential backoff) used all four attempts and sit in the failed set. The provider is back, but for a few minutes it still refuses about 1 call in 10; a job's next call after a refusal goes through. On-call runs await queue.retryJobs({ state: 'failed' }).

Once the queue is idle again, how many of the 300 jobs are back in the failed set?

Pick one

Question 4 of 6

A video platform renders thumbnails on a thumbnail queue served by one worker with concurrency 1; every job is added with attempts: 3 and no backoff. So that no thumbnail is ever lost, a teammate wrote this handler:

A creator's truncated video fails on every run. Its job was queued first, with 20 other videos behind it; those render on their first run.

What do you find when the worker finishes its 40th run (or goes idle, if that comes first)?

Code for this question
new Worker('thumbnail', async (job) => {  try {    return await renderThumbnail(job.data.videoId);  } catch (err) {    await thumbnails.add('thumbnail', job.data, { attempts: 3 });    return { requeued: true };  }}, { connection, concurrency: 1 });
Pick one

Question 5 of 6

A payout worker sends each payout job to a bank API; jobs have attempts: 3 and exponential backoff. At month end the bank answers 429 with Retry-After: 2 to the first four calls for every payout, then accepts. Goal: every payout is sent and none lands in the failed set.

Which handling of the 429 meets the goal?

Pick one

Question 6 of 6

A ticketing site issues e-tickets through an issue-ticket queue with one worker; jobs have attempts: 3 and removeOnFail: { count: 200 }. During an outage of the seat-map service, 1,200 jobs used up their attempts, one after another. After the fix, on-call replays the failed set with retryJobs.

Which jobs does the replay issue?

Pick one

Go deeper

Pro tips
  1. Copy the error's class into the failed job's reason. Throw new UnrecoverableError('CORRUPT_INPUT: …') with the cause in the message: the failed set keeps failedReason and the stack, so whoever replays can tell a fixed cause from an unfixed one. When: any handler that sorts failures (steps 3, 8).
  2. Size removeOnFail for your worst outage, not your normal day. The BullMQ docs suggest keeping "a much larger value of failed jobs" than completed ones. When: a queue whose failures can arrive in thousands during one outage (step 9).
Source
  • BullMQ docs, Stop retrying jobs (source docs/gitbook/patterns/stop-retrying-jobs.md, taskforcesh/bullmq branch master): https://docs.bullmq.io/patterns/stop-retrying-jobs · fetched 2026-10-18. "BullMQ will just move the job to the failed set without performing any retries, overriding any attempts settings used when adding the job to the queue" (step 3). Its passages on step 4 are in the assessment note below. Valid as of 2026-10-18.
  • BullMQ docs, Rate limiting, "Manual rate-limit" (source docs/gitbook/guide/rate-limiting.md): https://docs.bullmq.io/guide/rate-limiting · fetched 2026-10-18. Supports step 4; its passages are in the assessment note below. Valid as of 2026-10-18.
  • BullMQ docs, Retrying failing jobs (the curriculum row's source; docs/gitbook/guide/retrying-failing-jobs.md): https://docs.bullmq.io/guide/retrying-failing-jobs · fetched 2026-10-18. "If you do not specify a back-off function, the jobs will be retried without delay as soon as they fail"; "jobs will be moved at the end of our waiting list". Valid as of 2026-10-18.
  • BullMQ docs, Events and Workers (sources docs/gitbook/guide/events/README.md, docs/gitbook/guide/workers/README.md): https://docs.bullmq.io/guide/events and https://docs.bullmq.io/guide/workers · fetched 2026-10-18. "The events above are local for the workers that actually completed the jobs" (step 10). Valid as of 2026-10-18.
  • BullMQ docs, Retrying jobs (source docs/gitbook/guide/jobs/retrying-job.md): https://docs.bullmq.io/guide/jobs/retrying-job · fetched 2026-10-18. Supports step 12; its passages are in the assessment note below. Valid as of 2026-10-18.
  • BullMQ docs, Auto-removal of jobs (source docs/gitbook/guide/queues/auto-removal-of-jobs.md): https://docs.bullmq.io/guide/queues/auto-removal-of-jobs · fetched 2026-10-18. "A good practice is to keep a handful of completed jobs and a much larger value of failed jobs" (step 9, tip 2). Valid as of 2026-10-18.
  • AWS docs, Using dead-letter queues in Amazon SQS (the curriculum row's source): https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html · fetched 2026-10-18. "DLQs are useful for debugging your application because you can isolate unconsumed messages to determine why processing did not succeed"; "The maxReceiveCount is the number of times a consumer can receive a message from a source queue before it is moved to a dead-letter queue" (step 8, the term and the cap in SQS's words); "it is a best practice to always set the retention period of a dead-letter queue to be longer than the retention period of the original queue" (step 9, the same rule for SQS). Valid as of 2026-10-18.
  • AWS Builders' Library, Timeouts, retries, and backoff with jitter (the curriculum row's source): https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ · fetched 2026-10-18. "client errors should not be retried with the same request because they aren't going to succeed later" (step 1); "our solution is to limit the number of times that the client retries" (step 6). Backoff and jitter themselves are ds-retries-backoff-jitter's, cited there. Valid as of 2026-10-18.
  • BullMQ 6.3.9 installed source (node_modules/bullmq/dist/esm/): read for steps 2–4, 9, 10 and 12; what it says is in the assessment note below.

Related lessons

Checked against the tools and docs as of 2026-10-18.