Composing resilience policies: order is behaviour

Write the stack outside in; each layer sees only what the layer inside it hands back.

Before you read: predict

ExamHall's grading client writes each answer sheet's grade through the grading service:

At 0 ms the client calls gradeSheet('sheet-7'). The grading service takes 1,500 ms to answer the first call and 200 ms to answer each later one. Every call that reaches the service writes one grade for the sheet just before it answers.

What does the client get, and how many grades does sheet 7 have at 2,000 ms?

// grading-client.tsimport { wrap, retry, timeout, handleAll, ConstantBackoff, TimeoutStrategy } from 'cockatiel';import { grader } from './grader'; const perAttempt = timeout(1_000, TimeoutStrategy.Aggressive);const retries = retry(handleAll, { maxAttempts: 2, backoff: new ConstantBackoff(100) }); export const gradeSheet = (sheet: string) =>  wrap(retries, perAttempt).execute(() => grader.write(sheet));

The 30-second version

Timeout, retry, breaker, bulkhead and fallback each wrap the call, and wrap(a, b, c) puts a outermost. Read a stack from the outside in and ask what each layer sees: one attempt or the whole operation. The same five policies in another order give a different deadline, a different count and a different number of writes.

Write the stack outside in; each layer sees only what the layer inside it hands back.

How it works

Step 1 of 11

Outermost first. wrap(a, b, c).execute(fn) runs as a.execute(() => b.execute(() => c.execute(fn))): a sees what b returns, and only c sees the call itself.

Analogy: A master thatcher's office ringing homeowners the evening before the crew comes

At the thatcher's office	In code
"let it ring eight times, then hang up and dial again"	a per-attempt timeout under a retry (step 2)
"every homeowner rung before the crew's van leaves at seven"	one deadline over the whole evening (step 3)
hanging up does not stop the answerphone: a homeowner hears both messages, confirms twice, and two crew days are booked	an abandoned attempt still lands (steps 4–5)
saying the job number in every message, so the foreman sees one job	one key across every attempt (step 5)
"after five unanswered dials to a number, stop ringing it tonight": counting dials, or counting homeowners	a breaker inside or outside the retry (step 7)
a junior who posts a card as soon as one dial goes unanswered, so nobody rings back	a fallback inside the retry (step 9)

In real terms: before shipping a stack, say what each layer sees, one attempt or the whole operation, and what still runs after it gives up.

Breaks down when: the secretary hears the answerphone beep and knows a message landed; code that abandons an attempt hears nothing more about it.

Contrasting case: a job card handed over on the doorstep at the survey visit. Nothing is sent and nothing can time out, so there is nothing to wrap.

Examples, run for real

4.1 ExamHall EXH-4200: the grading client, as shipped and after MR !4500

EXH-4200: ExamHall's grading client retried each grade write with a 1,000 ms timeout per attempt. On a slow morning the grading service took 1,800 ms on a first call and 250 ms on later ones, and a sheet ended with two grades:

// grading-client.ts (as shipped)
const perAttempt = timeout(1_000, TimeoutStrategy.Aggressive);
const retries = retry(handleAll, { maxAttempts: 2, backoff: new ConstantBackoff(100) });
export const gradeSheet = (sheet: string) =>
  wrap(retries, perAttempt).execute(() => grader.write(sheet));

MR !4500 rewrites the stack outside in and mints one key per sheet before the retry, so every attempt at one sheet carries the same key. Both versions, under the same slow service:

4.1 EXH-4200 as shipped, slow service: caller `graded` at 1,350 ms · writes reaching the service 2 (at 1,350 ms, 1,800 ms) · grades stored 2
4.1 MR !4500, slow service: caller `graded` at 1,350 ms · writes reaching the service 2 (at 1,350 ms, 1,800 ms) · grades stored 1

As shipped, the second attempt answered and the abandoned first one still wrote (steps 2, 4). After the MR the same two writes reach the service, but they carry one key, so one grade is stored (step 5).

4.2 What the MR does not decide

How long each deadline is comes from the caller's budget and the service's latency, ds-partial-failure-timeouts' lane. Whether a write is safe to repeat without a key is ds-retries-backoff-jitter's question; how the service stores and replays a key is api-idempotency-keys'.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 8

A streaming service fetches a playback licence before each play:

From 20:00 the licence server answers every call with a 503 after 50 ms. Ten viewers press play one after another: each play calls getLicence once the previous play has its answer.

How many calls reach the licence server?

// licence-client.tsimport { wrap, retry, circuitBreaker, handleAll, ConstantBackoff, ConsecutiveBreaker } from 'cockatiel';import { drm } from './drm'; const retries = retry(handleAll, { maxAttempts: 2, backoff: new ConstantBackoff(100) });const breaker = circuitBreaker(handleAll, { halfOpenAfter: 30_000, breaker: new ConsecutiveBreaker(5) }); export const getLicence = (title: string) =>  wrap(retries, breaker).execute(() => drm.licence(title));

Question 2 of 8

A player loads a title's stream manifest from the CDN:

A viewer opens a title at 0 ms. The CDN's first call hangs for 10,000 ms and then returns the manifest; any later call returns it in 300 ms.

What does the viewer get, and how many calls reach the CDN?

// manifest-client.tsimport { wrap, retry, timeout, handleAll, ConstantBackoff, TimeoutStrategy } from 'cockatiel';import { cdn } from './cdn'; const overall = timeout(2_500, TimeoutStrategy.Aggressive);const retries = retry(handleAll, { maxAttempts: 2, backoff: new ConstantBackoff(200) }); export const getManifest = (title: string) =>  wrap(overall, retries).execute(() => cdn.manifest(title));

Question 3 of 8

A streaming home page loads a recommendations row:

In an outage the recommendations service answered every call with a 503 after 50 ms. Twenty home-page views, one after another, all showed the cached row, but the service got 20 calls and the breaker never opened. For the same 20 views the team wants: every view shows the cached row, the first view's failed call is tried again before it shows the cached row, and at most 5 calls reach the service.

Which line, in place of wrap(retries, breaker, cached), does all three?

// recs-client.tsimport { wrap, retry, circuitBreaker, fallback, handleAll, ConstantBackoff, ConsecutiveBreaker } from 'cockatiel';import { recs } from './recs'; const retries = retry(handleAll, { maxAttempts: 2, backoff: new ConstantBackoff(100) });const breaker = circuitBreaker(handleAll, { halfOpenAfter: 60_000, breaker: new ConsecutiveBreaker(5) });const cached = fallback(handleAll, 'cached row'); export const getRow = (viewer: string) =>  wrap(retries, breaker, cached).execute(() => recs.row(viewer));

Question 4 of 8

Before you write the stack, list your tests for licencePolicy, a streaming player's cockatiel policy for licence calls: each attempt is cut at 800 ms; the whole call ends by 2,000 ms; up to 2 retries, 100 ms apart, never on an open breaker; the breaker opens after 4 failed attempts in a row and tries again after 30,000 ms.

Name at least four tests, each with the licence server's behaviour and what the player should see.

Your answer (optional, stays on this page)

Question 5 of 8

Now write it, in 10 minutes, from a blank file that imports from cockatiel and exports licencePolicy.

Write the module in full.

Your answer (optional, stays on this page)

Question 6 of 8

In a mock interview you are asked: "Where does the timeout go relative to the bulkhead?" Three candidates each answered with a stack built from these two policies for a player's licence calls:

bulkhead(2, 0) lets 2 calls run at once, with no queue. The interviewer follows up: "The licence server hangs, and every call takes 3,000 ms. Viewers press play at 0, 200, 600 and 800 ms. Each viewer must get an answer within 500 ms, and the plays at 600 and 800 ms must both reach the server." No call reads the abort signal.

Whose stack survives the follow-up?

// licence-limits.tsimport { bulkhead, timeout, TimeoutStrategy } from 'cockatiel'; export const places = bulkhead(2, 0);export const perCall = timeout(500, TimeoutStrategy.Aggressive);

Question 7 of 8

A box office holds seats through the seat service:

bulkhead(2, 1) lets 2 executions run at once and 1 more wait for a place. The seat service fails each buyer's first call with a 503 after 100 ms and holds the seat on that buyer's second call, after 100 ms. Three buyers call holdSeat at 0, 200 and 400 ms.

When does buyer 3 get its hold?

// seat-client.tsimport { wrap, retry, bulkhead, handleAll, ConstantBackoff } from 'cockatiel';import { seats } from './seats'; const holds = bulkhead(2, 1);const retries = retry(handleAll, { maxAttempts: 1, backoff: new ConstantBackoff(500) }); export const holdSeat = (buyer: string) =>  wrap(holds, retries).execute(() => seats.hold(buyer));

Question 8 of 8

A payments service posts journal entries to the ledger:

The ledger answers every post with a 503 after 300 ms. At 0 ms the service calls postEntry('je-88').

How many posts reach the ledger in all?

// ledger-client.tsimport { wrap, retry, timeout, handleAll, IterableBackoff, TimeoutStrategy } from 'cockatiel';import { ledger } from './ledger'; const overall = timeout(2_000, TimeoutStrategy.Aggressive);const retries = retry(handleAll, { maxAttempts: 5, backoff: new IterableBackoff([200, 400, 800, 1_600, 3_200]) }); export const postEntry = (entry: string) =>  wrap(overall, retries).execute(() => ledger.post(entry));

Go deeper

Pro tips

Write the stack outside in and say what each layer sees. For each wrap(...), read it aloud from the left: "the overall deadline sees the operation; the retry sees attempts; the breaker sees attempts; the per-attempt timeout sees one call". When: reviewing any MR that adds or reorders a policy (steps 1, 11).
Hand the abort signal to the call. execute(({ signal }) => client.call(input, { signal })): an attempt a timeout gave up on then stops instead of landing late. When: any call wrapped in a timeout, above all a write (steps 4, 6).

tip 2 EXH-4200 as shipped, signal handed to the call: caller `graded` at 1,350 ms · writes reaching the service 1 (at 1,350 ms)

Source

cockatiel (the row's first source), readme, section "wrap(...policies)": https://github.com/connor4312/cockatiel · fetched 2026-10-14 from the raw readme.md. Teaching and the runs; every key here is executed. "Wraps the given set of policies into a single policy." and the readme's own equivalence of wrap(retry, breaker, timeout) with retry.execute(() => breaker.execute(() => timeout.execute(...))) (step 1); "In aggressive timeouts, we'll immediately throw a TaskCancelledError when the timeout is reached, in addition to marking the passed token as failed." (step 4). Valid as of 2026-10-14.
Amazon Builders' Library, "Timeouts, retries, and backoff with jitter" (the row's second source): https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/ · fetched 2026-10-14. Teaching only. "our view is that APIs with side effects aren’t safe to retry unless they provide idempotency" (step 5); "for low-cost control-plane and data-plane operations, our best practice is to retry at a single point in the stack" (the prerequisite's lane, linked). Valid as of 2026-10-14.
Google SRE book, "Handling Overload" (the row's third source): https://sre.google/sre-book/handling-overload/ · fetched 2026-10-14. Teaching only: "we implement a per-request retry budget of up to three attempts" (retry budgets are the prerequisite's lane; linked, no item here). Valid as of 2026-10-14.

Checked against the tools and docs as of 2026-10-14.

Predict first