Capacity from Little's law and the bottleneck
The slowest stage sets the pace. Everything let in beyond it only waits, and Little's law says for how long.
Predict first
Before you read: predict
The proctor console's live feed turns candidate events (a face leaving the camera, a tab switch) into proctor alerts. Each event passes three stages in turn: the ingest API (6 pods of 2 cores; 3 ms of CPU per event), the snapshot classifier (4 GPU workers, each classifying one event at a time for 10 ms) and an insert into Postgres (8 cores; 5 ms of CPU per insert). During a mock exam, events arrive at 2,000 a second for a whole minute.
Once the feed has settled, about how many alerts a second reach the proctors?
The 30-second version
Little's law: in a stable system, work in flight equals throughput times time in the system (L = λW), whatever the traffic's shape. A stage's capacity is its parallel units ÷ time per request; the smallest one, the bottleneck, caps throughput. Load beyond it only queues, so latency becomes L ÷ capacity. Size limits as capacity × unloaded time.
The slowest stage sets the pace. Everything let in beyond it only waits, and Little's law says for how long.
How it works
Step 1 of 11
In the kitchen
- loads in the shop, on average = loads finished an hour × hours each stays → L = λW (steps 1–2)
Analogy: A self-service launderette with eight washers
Each wash takes 40 minutes, and customers queue with their baskets.
| In the launderette | In the system |
|---|---|
| eight washers at 40 minutes a wash: at most 12 loads an hour, however long the line | a stage's capacity (step 3) |
| a second door, more baskets or more folding tables change nothing | only the bottleneck counts (step 4) |
| loads in the shop, on average = loads finished an hour × hours each stays | L = λW (steps 1–2) |
| a Saturday with 36 loads in the shop: each stays 3 hours, 40 minutes of it washing | W = L ÷ X (step 6) |
| a customer whose parking ticket runs out before a washer frees goes home | a timeout (step 9) |
| a sign-up sheet bringing one customer back every 5 minutes: each visit lasts 40 minutes, and the day still ends at the same time | admitting at X (steps 9–10) |
| regulars who wash, go home for an hour and come back: each one's round is the hour plus the wash | a boundary that includes the pauses (step 11) |
In real terms: find the stage with the fewest units per unit of work; its rate is the system's rate, and anything more in flight is a queue.
Breaks down when: washers never slow each other down. Postgres cores are shared by time-slicing, so extra concurrency slows every running query, not just the next in line.
Contrasting case: a launderette that is never busy. With no queue, W is just the wash time; Little's law still holds, but no stage is the limit yet.
Examples, run for real
4.1 ExamHall: the 09:00 start, re-executed
4.1 ECON-201 at 09:00: 2,400 candidates, 6 instances x pool 20, 2,000 ms acquire timeout; Postgres 8 cores, 20 ms CPU per paper query
4.1 bottleneck X = 8 cores ÷ 20 ms = 400 paper queries a second, fleet-wide
4.1 all at once: in Postgres 120 at a time · each query 300 ms · L = X × W = 400 × 0.3 = 120
4.1 all at once: served 840 · failed at the acquire timeout 1,560 · last served at 2,100 ms · longest pool wait of a served one 1,800 ms
4.1 admitted 8 every 20 ms (400 a second): served 2,400 · failed 0 · in Postgres at most 8 · each query 20 ms · last admitted at 5,980 ms · last served at 6,000 ms
The 300 ms is not slowness in Postgres: 120 queries share 8 cores, so W = 120 ÷ 400 (step 6). Admitting at X keeps every query at 20 ms and serves everyone by 6.0 s (steps 9–10).
4.2 ExamHall: "we doubled the instances" (EXH-1690, MR !1890)
At a practice sitting the API was doubled to 12 instances. Same model, with the timeout and without it:
4.2 change | conns | in Postgres | each query | served / failed (2 s timeout) | last served, no timeout
4.2 6 instances x pool 20 (today) | 120 | 120 | 300 ms | 840 / 1,560 | 6.0 s
4.2 6 instances x pool 50 | 300 | 300 | 750 ms | 900 / 1,500 | 6.0 s
4.2 12 instances x pool 20 | 240 | 240 | 600 ms | 960 / 1,440 | 6.0 s
4.2 400/s admission, 6 x 20 | 120 | 8 | 20 ms | 2,400 / 0 | 6.0 s
Every row ends at 6.0 s: 2,400 ÷ 400 (step 9). More instances or connections only move the queue into Postgres and stretch each query. The model does not refuse connections above max_connections 150; that refusal is pooling's step 8. MR !1890 admits exam starts at 400 a second instead.
4.3 Proof domain, catalog: Little's law on random traffic
A catalog's image-resize service with 3 workers, one simulated hour, two very different traffic shapes from one seeded generator:
4.3 random (exponential) gaps and resize times: 14,772 images · λ 4.102/s · W 1.405 s · λW 5.762 · L measured 5.762
4.3 near-regular (uniform ±50%) gaps and resize times: 14,513 images · λ 4.031/s · W 0.637 s · λW 2.567 · L measured 2.567
Burstier traffic waits longer, yet L = λW holds exactly for both (steps 1–2).
Try it
Answer first, then check. Nothing is saved.
Question 1 of 5
During a sale, a catalog's search pods send 10,000 stock lookups a second to the stock service. The service runs 16 worker threads; a lookup holds one thread for 4 ms, and lookups beyond 16 wait in its queue. Each lookup also spends 1 ms on the network first. The pods share one fleet-wide cap on lookups in flight; a lookup over the cap shows the cached stock flag instead.
Which cap answers the most lookups a second live, while each live lookup still takes 5 ms?
Question 2 of 5
At 10:00:00 a box office's waiting room starts letting fans through at 500 a second, 3,000 fans in all. The seat-hold service behind it has 10 workers; a hold takes one worker 40 ms, and waiting holds are served in arrival order. Nothing times out.
About when does the last fan's hold finish?
Question 3 of 5
A payment provider's chargeback desk has three analysts. They keep 12 disputes open at any time and decide 3 a week, month after month, while hundreds more wait to be opened; each dispute is about one week of one analyst's work. The lead wants a dispute to take 2 weeks from opening to decision, without hiring and while still deciding 3 a week.
Which change gets there?
Question 4 of 5
A nightly job reprices 60,000 products and must finish within 10 minutes, but it runs past that. Twenty job workers each take one product at a time: a read from the product database (8 cores, 2 ms of CPU), one call to the currency API, which starts at most 80 calls a second per API key and answers in 5 ms, then a write to the search index (4 nodes, 10 ms per write, one write per node at a time). Any part can change: the worker count, the database's size, the number of search nodes, or how many products go in one currency call.
Which one change gets the job under 10 minutes?
Question 5 of 5
A chat app's presence panel keeps itself fresh: each open client asks the presence service who is online and, 300 ms after the answer arrives, asks again. One service instance has 4 workers; an answer takes one worker 200 ms, and waiting requests queue in arrival order. The team is deciding whether an instance needs a cap on open clients, with the rest sent to another instance.
What is the most open clients one instance can hold while every answer still takes 200 ms?
Go deeper
Pro tips
- Name the bottleneck and its unit before sizing anything: cores, connections, partitions, a provider's rate limit. Every other box is sized from it, and saying so is a senior signal (curriculum pro tip). When: the first minute of a capacity question (steps 3–4).
- Check Little's law on your own dashboards. In-flight gauge ≈ throughput × mean latency. If they disagree, one metric is measured at a different boundary, for example latency that starts after the pool wait. When: before trusting a latency graph during an incident (steps 1–2).
Source
- John D. C. Little, "A Proof for the Queuing Formula: L = λW", Operations Research 9(3):383–387, 1961 (doi 10.1287/opre.9.3.383). Abstract, read 2026-09-29 through the Crossref record https://api.crossref.org/works/10.1287/opre.9.3.383 (saved as
sources/little1961-crossref.json, sha256 680dfaa3…98fe): "It is shown that, if the three means are finite and the corresponding stochastic processes strictly stationary, and, if the arrival process is metrically transitive with nonzero mean, then L = λW." The publisher page answered 403, so the article body was not read. Supports steps 1–2 (means only, a stationary system). Valid as of 2026-09-29. - Netflix concurrency-limits, README: https://github.com/Netflix/concurrency-limits (raw README fetched 2026-09-29, saved as
sources/netflix-readme.md, sha256 abe31732…9ba6): "we should be thinking in terms of concurrent requests where we apply queuing theory to determine the number of concurrent requests a service can handle before a queue starts to build up, latencies increase and the service eventually exhausts a hard limit such as CPU, memory, disk or network. This relationship is covered very nicely with Little's Law whereLimit = Average RPS * Average Latency." Supports steps 7–8. The library's adaptive algorithms (Vegas, Gradient) are not taught or run here. Valid as of 2026-09-29.
Related lessons
Checked against the tools and docs as of 2026-09-29.