Scaling queue consumers: concurrency, partitions and ordering

Consumers beyond the ordering units sit idle, and the busiest unit, not the average one, decides when the backlog is gone.

Before you read: predict

ExamHall grades each submitted attempt from a Kafka topic. When ECON-301 closes, 1,800 grade jobs land on it, keyed by attempt ID, so its 6 partitions hold 300 jobs each. A classic consumer group of 4 grader instances reads the topic; each grader works on one job at a time, and a job takes 300 ms.

Counted from the close, when does the last job finish?

The 30-second version

A backlog drains in backlog × job time ÷ jobs running at once. Ordering caps that count: a Kafka partition has one consumer in a classic group, an SQS FIFO group one message in flight. One key's work never splits, so the busiest unit sets the finish.

Consumers beyond the ordering units sit idle, and the busiest unit, not the average one, decides when the backlog is gone.

How it works

Step 1 of 11

Drain time = backlog × time per job ÷ jobs running at once. It is Little's N ÷ X, with X = those jobs ÷ the job time.

Analogy: A high-street copy shop's row of numbered copiers, the evening before a deadline

In the copy shop	In the system
the counter sends every ticket with one customer's name to the same copier	a key picks the partition (steps 3, 6)
one operator per copier, one job at a time	one consumer per partition (step 4)
a temp hired for the rush, with no copier of her own, waiting by the door	consumers beyond the partition count (step 4)
eight copiers, six operators: two walk between two machines and leave last	partitions dealt whole (step 5)
a law firm's forty bundles, all under the firm's name, keep copier 3 busy after the others stop	the busiest unit sets the finish (steps 6–7)
writing the case name on each ticket instead, because only one case's bundles must stay in order	the narrowest key (step 8)
any free copier taking the next page from any pile	a share group (step 10)

In real terms: count the units that may run at once and find the fullest one; that unit, not the average, is your drain time.

Breaks down when: the counter can see which copier is free. A partitioner hashes the key and never looks at load, so a hot key stays where it landed.

Contrasting case: a stack of identical flyers. No page order matters, so any copier takes any sheet; work with no ordering need has no ordering unit, and only workers and the downstream limit it.

Examples, run for real

Both run on the virtual clock, which also asserts the curriculum hook's numbers.

4.1 ExamHall: ECON-201's grading queue (EXH-4240, MR !4540)

ECON-201 closes at 11:00 and enqueues 2,400 grade-exam jobs of 200 ms each (a scenario value). Order matters per attempt, because a regrade must follow its first grading, so the narrowest correct key is the attempt ID. The grader first keyed jobs by exam ID:

4.1 ECON-201 at 11:00: 2,400 grade-exam jobs, 200 ms each
4.1 BullMQ, 4 workers x concurrency 10, no key               40 at once · drained at 12 s
4.1 Kafka, keyed by exam ID, 6 partitions, 6 graders          busy 1 of 6 · drained at 8 min (EXH-4240)
4.1 Kafka, keyed by attempt ID, 400 a partition, 6 graders   busy 6 of 6 · drained at 1 min 20 s (MR !4540)

Keyed by exam, one partition holds every job, and one grader works while five wait (steps 6–7). Keyed by attempt, six run at once (step 4). BullMQ keeps no per-attempt order, so its 40 slots all run (step 2).

4.2 Media streaming: a share group on two kinds of work

Thumbnails need no order; chapter-marker edits on a video do. Both backlogs sit on 4 partitions, with each video's edits arriving in bursts:

4.2 1,600 thumbnail jobs, 4 partitions, 1 s each · classic group of 16: busy 4, drained at 6 min 40 s · share group of 16: drained at 1 min 40 s
4.2 1,600 chapter-marker edits on 40 videos, 4 partitions · classic group of 16: drained at 6 min 40 s, out of order 0 · share group of 16: drained at 1 min 40 s, out of order 1,200

The share group lifts the partition cap for both, and pays for it only where order matters (step 10).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

A video platform's transcode topic has 12 partitions and is keyed by channel ID; a classic consumer group of 12 transcoders reads it, each working on one job at a time, 24 s per job. Tonight's backlog is 1,140 jobs. A film studio's channel uploaded 300 of them, and that channel sits on partition 7; the other 840 come from hundreds of small channels and sit evenly on the 12 partitions, 70 on each. To finish sooner, the team runs 24 transcoders instead of 12.

Counted from now, when does the backlog drain?

Question 2 of 6

A video platform applies caption edits from an Amazon SQS FIFO queue read by 24 consumers, each receiving one message per call; an edit takes 500 ms to apply. Each edit overwrites one caption track, so a track's edits must apply in the order they were made. Tonight's backlog is 4,800 edits from 37 channels. A news channel sent 1,200 of them: 600 to its live broadcast (6 language tracks, 100 edits each) and 600 to 30 recorded videos (2 tracks, 10 edits each). Each of the other 36 channels sent 100: 5 videos, 2 tracks, 10 edits each.

Which message group ID empties the queue soonest with every track's edits applied in order?

Question 3 of 6

In an interview you answered: "Each streamer's events must stay in order, so I key them by streamer ID; 16 partitions and 16 consumers share the work." The interviewer's follow-up: "One broadcaster account runs 8 live streams at once and sends 4,000 of the 10,000 events in tonight's backlog; 600 single-stream streamers send 10 each." Each event updates one live stream's viewer counters and takes 200 ms, and the backlog takes about 15 minutes.

Which answer brings it under 5 minutes with each stream's events still applied in order?

Question 4 of 6

A video platform's playback-events topic has 6 partitions, keyed by viewer ID, and a classic group of 6 consumers applies the events, one at a time, 500 ms each. Tonight's backlog holds 4,800 events from 600 viewers, and each viewer's 8 events (play, pause, seek and so on) sit next to each other in its partition. A viewer's events must be applied in the order they happened.

Which change empties the backlog soonest with every viewer's events still in order?

Question 5 of 6

A ticketing site applies seat-map updates from an Amazon SQS FIFO queue whose message group ID is the event ID. A stadium concert went on sale, and tonight's backlog holds 8,000 updates: 6,000 for the concert and 1,000 for each of 2 smaller events. 40 consumers poll the queue, each receiving up to 10 messages per call and applying them one after another, 200 ms per update.

When is the whole backlog applied?

Question 6 of 6

A chat app's search indexer reads a Kafka topic with 4 partitions, keyed by room ID, as a classic group of 2 consumers: one reads partitions 0 and 2, the other partitions 1 and 3. Each consumer runs 8 threads and picks a message's thread by its room ID, so each room's messages are indexed in order; a thread indexes one message at a time, 100 ms each. On a busy evening the backlog holds 12,000 messages from 1,200 rooms, 10 each: 4,800 on partition 0, 3,200 on partition 2 and 2,000 on each of partitions 1 and 3, and each consumer's threads get equal shares of its rooms.

When is the whole backlog indexed?

Go deeper

Pro tips

Say the minimum out loud. "Jobs at once is the smallest of workers × concurrency, partitions or groups, and what the downstream accepts; then I find the busiest unit." Naming the cap before proposing more consumers is the senior answer to "we added consumers and nothing changed". When: the first minute of a queue-throughput question (steps 1–2).
Alert on the age of the oldest job, not the queue length (curriculum pro tip): 10,000 jobs draining at 2,000 a second are fine; 50 stuck for 20 minutes are an incident. When: setting a queue's alerts; per-unit age also exposes one hot partition behind a short total. Verify: not executed in this lesson; it is the curriculum row's tip, and scaling on backlog age belongs to scale-autoscaling-cold-starts.

Sources

Apache Kafka 4.3 Javadoc, KafkaConsumer: https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html. "This is achieved by balancing the partitions between all members in the consumer group so that each partition is assigned to exactly one consumer in the group. So if there is a topic with four partitions, and a consumer group with two processes, each process would consume from two partitions." Supports steps 3–5 and the model's assignment. Also, under "One Consumer Per Thread": "The number of total threads across all processes will be limited by the total number of partitions." and, under "Decouple Consumption and Processing": "Guaranteeing order across the processors requires particular care as the threads will execute independently an earlier chunk of data may actually be processed after a later chunk of data just due to the luck of thread execution timing." Supports steps 4 and 9 (choosing the thread by key is this lesson's executed model, not the Javadoc's wording). Valid as of 2026-10-15.
Apache Kafka 4.3 Javadoc, KafkaShareConsumer: https://kafka.apache.org/43/javadoc/org/apache/kafka/clients/consumer/KafkaShareConsumer.html. "Unlike consumer groups, share groups balance the partitions between all members of the share group permitting multiple consumers to consume from the same partitions. This gives more flexible sharing of records than a consumer group, at the expense of record ordering." Supports step 10. Valid as of 2026-10-15.
Apache Kafka 4.2.0 release announcement, 17 February 2026: https://kafka.apache.org/blog/2026/02/17/apache-kafka-4.2.0-release-announcement/. "Kafka Queues (Share Groups) is now production-ready" and "KIP-932: Queues for Kafka Introduces share groups, a new cooperative consumption model where multiple consumers can concurrently process records from the same partitions with individual acknowledgment and delivery counting". Supports step 10's "Kafka 4.2". Valid as of 2026-10-15.
Apache Kafka 4.3 documentation, Introduction: https://kafka.apache.org/43/getting-started/introduction/. "Events with the same event key (e.g., a customer or vehicle ID) are written to the same partition, and Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written." Supports steps 3 and 6. Valid as of 2026-10-15.
Amazon SQS Developer Guide, "FIFO queue delivery logic in Amazon SQS": https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html. "Allows other consumers to process messages from different message group IDs concurrently." and "However, you can't receive additional messages from the same message group ID in subsequent requests until: The currently received messages are deleted, or They become visible again (for example, after the visibility timeout expires)." Supports steps 3 and 11 and the model's gate. Valid as of 2026-10-15.

Checked against the tools and docs as of 2026-10-15.

Predict first