Circuit breakers and bulkheads

A breaker stops calling a failing dependency for a while, then tests it with one call; a bulkhead stops one slow dependency taking everyone's capacity.

Before you read: predict

At 08:55:00 on ECON-201 day the face-match vendor starts answering every call with a 503, and keeps doing so for the next 60 s. Candidates check in at a steady 5 a second, and each check-in makes one face-match call through the check-in API's breaker (opossum):

How many face-match calls reach the vendor in those 60 s?

const faceMatch = new CircuitBreaker(callVendor, {  timeout: 3000,                  // a call still unanswered after 3 s counts as failed  errorThresholdPercentage: 50,   // opens when more than half of recent calls failed,  volumeThreshold: 10,            //   once at least 10 calls are in the window  rollingCountTimeout: 10000,     // "recent" means the last 10 s  resetTimeout: 30000,});

The 30-second version

A circuit breaker counts a dependency's recent failures and, past a threshold, opens: calls fail at once without reaching it, so the caller needs a fallback. After a set wait, one trial call goes through: success closes the breaker, failure reopens it for another wait. A bulkhead caps each dependency's share of the caller's capacity and turns the excess away.

A breaker stops calling a failing dependency for a while, then tests it with one call; a bulkhead stops one slow dependency taking everyone's capacity.

How it works

Step 1 of 14

A call to a slow dependency holds a request slot, socket or worker of the caller until it is answered or cut off.

In the kitchen

an agent on hold to a ferry line that has stopped answering serves no one else → a slow call holds the caller's slot (step 1)

Analogy: A high-street travel agency's booking desk

Six agents serve the queue, and every booking needs a phone call to a supplier: the ferry line, the hotels, the rail desk.

At the booking desk	In the system
an agent on hold to a ferry line that has stopped answering serves no one else	a slow call holds the caller's slot (step 1)
the rule: at most two agents on hold to the ferry line; a third ferry customer gets "details now, confirmation by email"	a bulkhead that turns the extra away (steps 2–3)
a third agent who stands at the desk until a ferry line frees up	waiting for a place still holds the slot (step 3)
several unanswered ferry calls in a short spell: "ferry: do not ring, take details" on the whiteboard	the breaker opens (steps 6, 8)
twenty minutes later, one agent rings once	half-open: one trial (step 9)
answered, the board is wiped; unanswered, another twenty minutes	closed, or open again (step 10)
the board mends nothing at the ferry company, and customers still leave with their slip	a breaker protects the caller and needs a fallback (steps 12–13)

In real terms: cap what each dependency may tie up, and stop calling one that keeps failing, with a planned answer for the caller meanwhile.

Breaks down when: an agent can read the ferry company's status page; a caller sees only its own calls. A customer can wait on the sofa without tying up an agent, but a request waiting inside a process always holds something.

Contrasting case: a ferry line that answers "fully booked" at once. No agent is ever on hold, so the two-agent rule never applies; only the whiteboard stops the wasted calls (step 14).

Examples, run for real

4.1 ExamHall EXH-2180: a slow vendor, before and after MR !2480

EXH-2180: from 08:55 the face-match vendor answered every match after 20 s. The check-in API called it with only a 30 s timeout, so each of the 600 candidates checking in over the next two minutes waited 20 s. MR !2480 wraps the call in a breaker with a 3 s timeout that opens when over half of at least 10 calls in 30 s fail, and whose fallback sends the candidate to the proctor's manual ID check:

4.1 before: 30 s timeout, no breaker   check-ins 600 · answered after the vendor's 20 s: 600 · sent to the manual check 0 · calls that reached the vendor 600
4.1 after MR !2480                     breaker opened at 08:55:06 · sent to the manual check at once 566, after the 3 s timeout 34 · trial calls 4, each cut off at 3 s · calls that reached the vendor 34

The 3 s timeout made each slow reply a failure (steps 4–5), so the breaker opened six seconds in (step 6). From then on, check-ins went straight to the manual check (steps 8, 13), and each trial was cut off at 3 s and reopened the breaker (steps 9–10).

4.2 Proof domain, checkout: one slow dependency, two routes

A checkout pod has room for 12 requests at once. Tax quotes call the tax-rates service; payments call the payment provider. From 15:00:00 the tax service answers after 10 s, and every second brings 2 tax quotes and 4 payments:

4.2 no bulkhead                                      payments answered 503: 46 of 80 (from 15:00:05.5) · tax quotes from the service 24, estimated 0, answered 503 16
4.2 bulkhead: 3 tax calls; a 4th gets the estimate   payments answered 503:  0 of 80 · tax quotes from the service 6, estimated 34, answered 503 0

Without a bulkhead, waiting tax quotes took all 12 places and payments were turned away (step 1). With it, tax calls can hold 3 places at most, the rest get the estimate at once, and payments keep their places (steps 2–3).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 5

The shop's checkout API charges every order through the payment provider:

From 14:00:00 the provider accepts connections but sends no reply at all, for the next 120 s. Shoppers place 2 orders a second.

What do the 240 shoppers who order in those 120 s get back?

const charge = new CircuitBreaker((order) => provider.charge(order), {  timeout: 5000, errorThresholdPercentage: 50, volumeThreshold: 10,  rollingCountTimeout: 10000, resetTimeout: 30000,});app.post('/checkout', async (req, res) => {  try { res.status(201).json(await charge.fire(req.body)); }  catch { res.status(502).json({ error: 'payment failed' }); }});

Question 2 of 5

A product-page pod serves review panels and product pages; Sema is async-sema's semaphore:

From 10:00:00 the reviews service answers each call only after 8 s. Every second, browsers ask this pod for 3 review panels and 5 product pages. From 10:00:06 product pages start failing with 503, although reviews calls are capped at 4. The slowdown lasts 30 s. Line 4 stays the pod's one memory cap, shared by every route, whatever number it holds.

Which line lets the reviews slowdown take the product pages down?

1  const reviewsSlots = new Sema(4);                               // 4 reviews calls at once2  let inFlight = 0;3  http.createServer(async (req, res) => {4    if (inFlight >= 20) return res.writeHead(503).end();          // the pod has memory for 20 requests at once5    inFlight++;6    try {7      if (isReviews(req)) { await reviewsSlots.acquire();8        try { send(res, await reviews.get(idOf(req))); } finally { reviewsSlots.release(); } }9      else send(res, await inventory.get(idOf(req)));              // product page: stock and price10   } finally { inFlight--; }11 });

Question 3 of 5

A support chat translates each incoming message through this breaker; while the call fails, the fallback shows the message untranslated:

One message arrives every 2 s, from 12:00:00. The translation service answers 503 to every call until 12:00:40, and answers normally from then on.

When is a message first shown translated again?

const translate = new CircuitBreaker((msg) => translator.translate(msg), {  timeout: 2000, errorThresholdPercentage: 50, volumeThreshold: 5,  rollingCountTimeout: 10000, resetTimeout: 25000,});translate.fallback((msg) => msg.text);   // show the original

Question 4 of 5

A video service checks a viewer's licence once per stream start, through this breaker; while it is open, the fallback starts the stream on the viewer's cached licence:

Last night from 03:00 to 03:05 the licence server answered 503 to every check, and the breaker stayed closed the whole time: each stream start sent a check that failed. At that hour one stream starts every 4 s; by day 10 start each second, and about 1 check in 50 fails.

Which one change makes the breaker open within a minute of a night outage like that one, and keeps it closed through a normal day?

const licence = new CircuitBreaker(checkLicence, {  timeout: 3000, errorThresholdPercentage: 50, volumeThreshold: 10,  rollingCountTimeout: 10000, resetTimeout: 30000,});

Question 5 of 5

A ticket shop's seat map calls the seat-hold service each time a fan clicks a seat, 5 clicks a second. From 19:00:00 to 19:01:00 the overloaded seat-hold service answers 503 at once to every call; from 19:01:00 it is healthy.

Which one tool around the seat-hold call sends it at most 20 calls between 19:00:00 and 19:01:00, and has seats being held again by 19:01:40?

Go deeper

Pro tips

Alert when a breaker opens, and graph its state beside the dependency's error rate. An open breaker answers every caller with the fallback, so the outage it hides shows in no error log. When: adding a breaker to any call (steps 8, 13). Status: DOC (the curriculum's pro tip; Fowler: "Usually you'll also want some kind of monitor alert if the circuit breaker trips"); not run in this session.

Source

Library behaviour can change between versions, so the opossum and async-sema claims are valid as of 2026-10-01 for the pinned versions.

Martin Fowler, CircuitBreaker, 6 March 2014, the curriculum row's first source: https://martinfowler.com/bliki/CircuitBreaker.html (fowler-circuit-breaker.txt). "Once the failures reach a certain threshold, the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all" (steps 6, 8); "Asked to call in the half-open state results in a trial call, which will either reset the breaker if successful or restart the timeout if not" (steps 9–10); "if you have many callers on a unresponsive supplier, then you can run out of critical resources" (step 1).
Microsoft Azure Architecture Center, Circuit Breaker pattern: https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker (azure-circuit-breaker.txt). Half-Open: "A limited number of requests from the application are allowed to pass through and invoke the operation"; "If any request fails, the circuit breaker assumes that the fault is still present, so it reverts to the Open state. It restarts the time-out timer" (steps 9–10); "an application might temporarily degrade its functionality, invoke an alternative operation … or report the exception to the user" (step 13).
Microsoft Azure Architecture Center, Bulkhead pattern: https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead (azure-bulkhead.txt). "a consumer that calls multiple services might be assigned a connection pool for each service. If a service begins to fail, it only affects the connection pool assigned for that service" (step 2).
Google SRE book, ch. 22, Addressing Cascading Failures, the row's fourth source: https://sre.google/sre-book/addressing-cascading-failures/ (sre-cascading.txt). "When a frontend sends an RPC to a backend server, the frontend consumes resources waiting for a reply" (step 1); "Usually, if the queue is full, the server will reject new requests" (step 3).
AWS Builders' Library, Timeouts, retries, and backoff with jitter (Marc Brooker), the row's third source: requested at https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/, which redirects to https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter; that page serves an empty shell to curl, so it was read through Chromium 153.0.8010.12 (builders-timeouts.txt, with a Via line). "Circuit breakers, where calls to a downstream service are stopped entirely when an error threshold is exceeded"; "circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery" (step 11).
opossum 10.0.0 README and lib/circuit.js: https://raw.githubusercontent.com/nodeshift/opossum/v10.0.0/README.md and https://raw.githubusercontent.com/nodeshift/opossum/v10.0.0/lib/circuit.js (opossum-readme.txt, opossum-circuit-js.txt; the second is byte-identical to the installed copy). "Once in the halfOpen state, the next time the circuit is fired, the circuit's action will be executed again. If successful, the circuit will close … If the action fails or times out, it immediately re-enters the open state"; its JSDoc for volumeThreshold: "the minimum number of requests within … the rolling statistical window that must exist before the circuit breaker … can open" (steps 7, 9–10).
async-sema README (the repository has no 3.1.1 tag, so the default branch was read; its text is byte-identical to the README in the installed 3.1.1 package): https://raw.githubusercontent.com/vercel/async-sema/main/readme.md (async-sema-readme.txt). tryAcquire(): "Attempt to acquire a token from the semaphore, if one is available immediately. Otherwise, return undefined.".
**Michael Nygard, Release It!, 2nd ed.**, the row's second source, is a book that could not be fetched as text; Fowler names it as the pattern's origin. It keys nothing here.

Checked against the tools and docs as of 2026-10-01.

Predict first