Overload protection
Protect useful completions by deciding what enters expensive service before an overloaded queue decides for you.
Predict first
Before you read: predict
Five proctor refreshes arrive before any service completion. The fixture uses a two-slot admission check.
For how many does the backend emit 503 before the virtual clock advances?
The 30-second version
Goodput counts successful responses that remain useful. Expired queued requests can obstruct new work. Bound admission, counting active and queued jobs in this fixture, and reject excess before expensive processing. A waiting room holds users outside the protected service. Rate limits constrain arrivals over time; load shedding protects current capacity. S1 S5
Protect useful completions by deciding what enters expensive service before an overloaded queue decides for you.
How it works
Step 1 of 7
Analogy: A climbing gym's entry marshal
Climbers need equipment checks before entering the roped floor. An uncontrolled inspection backlog can crowd the area while people miss their booked session. The marshal admits only the people the inspection area can handle, asks others to wait outside, and turns away arrivals whose session cannot be reached. Moving the queue outside does not extend anyone's booking.
| Climbing gym | Service mechanism |
|---|---|
| Equipment inspection area | Expensive backend work |
| Limit on people inside | Admission bound |
| Waiting outside the roped floor | External waiting room |
| Missed booked session | Result beyond its useful deadline |
In real terms: keep a backlog from spending the capacity intended for useful work. Breaks down when: people can negotiate another booking; a request deadline must not silently restart. Contrasting case: a quiet gym may enforce a member's daily entry allowance even without crowding. Likewise, a client quota and an overload signal answer different questions. S5
The waiting area also needs a policy for expiry and excess demand. It is not an unlimited storage promise. A rejection still consumes some networking and application resources; protection has a finite operating range. Selecting a bound requires testing the actual service, because request costs can differ. This lesson's identical-cost jobs isolate admission decisions; they do not predict a production safe limit. S6
Examples, run for real
4.1 ExamHall: expired refreshes block useful work
EXH-1640, Expired proctor refreshes occupy the service queue. A local HTTP backend uses one virtual worker. Each job consumes one tick; deadlines are absolute and inclusive. The fixture intentionally finishes expired work instead of cancelling it. Actual HTTP requests remain open until their modelled completion; explicit arrival acknowledgements remove races. S2
The low-load run receives two requests at tick zero with deadline two, then two at tick two with deadline four. The overload run receives eight in the first wave and the same later wave. Count useful completions through tick four, including completions at that tick. Both runs have the same worker and observation window.
example useful completions: low=4 unbounded=2 bounded=4
Expired first-wave jobs occupy later worker turns in the unbounded run. This is a collapse in useful completions, even though service continues. It demonstrates wasted FIFO work, not a measurement of CPU thrashing. MR !1840, Bound proctor admission before expensive refresh work, limits total inside to two. Excess arrivals receive early errors; capacity is available when the later wave arrives. S1
4.2 Catalog: waiting does not restart a deadline
A catalog gateway initially holds three requests outside a backend that permits two inside. Every request has the original absolute deadline one. The gateway admits another only after capacity frees and discards waiting entries whose deadlines have passed. Through tick three, only one result is useful. A request admitted exactly at its deadline cannot finish a one-tick job in time; preserving the deadline exposes that lost opportunity.
waiting room admission layout: inside=2 outside=3; absolute-deadline useful=1
The HTTP transport is real; the waiting-room controller and service-cost scheduler are deliberately small executed models. No virtual tick represents measured wall-clock latency. A real waiting room protects the origin by withholding arrivals; its user experience and admission policy require separate design. S3
Try it
Answer first, then check. Nothing is saved.
Question 1 of 5
Six catalog refreshes arrive before the first completion. The handler is if(full) job.reject=true; queue.push(job); start(). Every job takes one tick and has deadline nine.
By tick six, how many expensive jobs have started?
Question 2 of 5
Five catalog requests with deadline nine enter an external FIFO waiting room at tick zero. The controller permits one inside the backend and releases the next immediately after each one-tick completion.
Just after the first completion and replacement admission, how many remain outside waiting?
Question 3 of 5
Three refreshes arrive at tick zero and two at tick one after one completion. A service has one backend slot. The fixture compares a full-capacity shedder, a two-arrivals-per-tick limiter, and open admission.
How many 503s does the run record through tick one for the capacity shedder?
Question 4 of 5
A catalog worker has a three-slot bound. Six requests arrive at tick zero with deadline two; three arrive at tick three with deadline six, after that tick's completion. Deadlines are inclusive. The trace records [useful completions, successful jobs completed after their deadline].
Which tuple does the bounded FIFO candidate record through tick six?
Question 5 of 5
Three requests arrive while completion is held. The candidate sets limit=1 and hint="2", then runs if(full){ reply(503,{Retry-After:hint}); return; } enqueue(). After the test collects all responses, it records [503s before the first completion, responses carrying Retry-After, jobs started before that completion].
Which tuple does the candidate produce?
Go deeper
Pro tip
Record admitted, rejected and useful-success counts separately during an overload test. One combined error percentage obscures which population a change affected. Keep this diagnostic view alongside the offered-load curve. AWS Builders' Library, Visibility, read 2026-09-30.
Sources
Fetched and read 2026-09-30. Fixture observations are reproducible in mechanisms.mjs; sources justify the general mechanisms, not the authored scenario inputs.
- S1: David Yanacek, Using load shedding to avoid overload, AWS Builders' Library: goodput, wasted work, early rejection and overload testing.
- S2: Mike Ulrich, Addressing Cascading Failures, Google SRE: queue management, deadlines and overload.
- S3: Cloudflare, Waiting Room: holding excess users away from the origin.
- S4: RFC 9110 503 and Retry-After: temporary overload and the response field.
- S5: Nottingham and Fielding, RFC 6585 §4: rate limiting and 429; saved as
sources/rfc6585.txt, read 2026-09-30. - S6: Alejandro Forero Cuervo, Handling Overload, Google SRE: quotas, utilization signals, rejection cost and protection limits.
Related lessons
Checked against the tools and docs as of 2026-09-30.