Tail latency at scale: fan-out, hedging, deadlines

A page that waits for every call inherits the worst of them: fan-out multiplies the chance of meeting somebody's tail.

Before you read: predict

A payments ledger's statement page asks 40 account shards for their entries in parallel and renders once all 40 have answered. A shard answers in 5–15 ms, except 1 call in 100, at random, which takes 1 s; a slow call on one shard says nothing about the others.

To the nearest whole percent, what share of statement pages take 1 s or more?

The 30-second version

A request that waits for n parallel calls is as slow as the slowest. If each call is slow with chance p, independently, 1 − (1 − p)^n of requests are slow, so a rare leaf tail becomes a common page tail. Cut fan-out or thin each leaf's tail, hedge to replicas after the typical wait, and give the fan-out one deadline.

A page that waits for every call inherits the worst of them: fan-out multiplies the chance of meeting somebody's tail.

How it works

Step 1 of 13

Fan-out: a request split across n leaves in parallel, and waiting for all of them, is as slow as its slowest leaf.

In the kitchen

the procession waits for the last float → a request waits for its slowest leaf (step 1)

Analogy: A town carnival procession

The procession leaves its assembly field only once every float has reported to the marshal.

On the field	In a system
the procession waits for the last float	a request waits for its slowest leaf (step 1)
each float breaks down one year in a hundred, yet with forty floats the start is often late	fan-out multiplies the tail (steps 2–3)
once a float is later than nineteen in twenty usually are, send the spare tractor	a hedge after the typical wait (steps 9–10)
leave at 14:00 with the floats that are there; the rest stand down	one deadline, a partial answer (steps 12–13)

Breaks down when: a crowd still cheers a float that arrives five minutes late; a reply after its deadline is thrown away.

Contrasting case: a single marching band is late only when the band itself is late: no fan-out, no multiplication (step 2 with n = 1).

In real terms: count the calls a request waits for, and design for the share of requests that meet at least one slow call, not for each call's percentile.

Examples, run for real

4.1 ExamHall EXH-4220: the proctor dashboard waits for six widgets

ExamHall's proctor dashboard loaded six widgets in parallel and rendered once all six had answered. Each widget's call took 1 s on 1 call in 100.

4.1 before: 6 widgets, all awaited · 6025 of 100000 dashboard loads take 1 s (6.0%) · 1 - 0.99^6 = 5.9% · the same widgets at fan-out 100: 63.4%
4.1 MR: each widget behind its own Suspense boundary · loads where the whole dashboard waits on a slow widget 0 of 100000 · a given widget shows its fallback past 1 s on 1015 of 100000 loads (1.0%)

Six calls at 1% each make about 1 dashboard in 17 slow (steps 1–3). MR !4520 renders each widget behind its own Suspense boundary, so the dashboard no longer waits for all six: a slow widget delays only itself.

4.2 Racing a stalled replica

An exam-paper store answers a read in 2–8 ms, except 1 call in 100 that stalls for 100–500 ms.

4.2 one replica: read p99 101 ms · a copy to a second replica after 8 ms: read p99 10 ms, 1.0% more calls

One replica's stalls set its p99; a copy raced against a stall takes them out of it, and every copy is one more call (steps 9–10).

4.3 One deadline across twelve shards

ExamHall's results page asks 12 shards with one 200 ms deadline; 1 call in 50 waits 1 s in its shard's queue.

4.3 1000 loads: pages rendered at 200 ms with a shard missing 206 · shard work after the deadline: shards checking the deadline 0 ms · shards keeping their own 2 s timeout 13860 ms

The page answers on time with a gap instead of waiting (steps 12–13); shards that check the deadline drop work nobody will read.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

A video service's home page asks 60 catalogue shards in parallel and renders once all have answered. A shard call takes 5–15 ms, except 1 call in 50, at random, which takes 1 s; shards are slow independently, and today 70% of home pages take 1 s.

Which single change brings the share of home pages taking 1 s under 10%?

Question 2 of 6

A payments ledger reads each balance from one of three replicas. A call takes 5–15 ms, except 2 calls in 100, at random on any replica, that stall for 200–1,000 ms. When two copies of a read are out, the first reply is used and the other copy is cancelled. The team wants the read p99 under 50 ms while sending at most 10% more calls.

Which policy meets both?

Question 3 of 6

A chat service's history API runs on 4 servers, each with its own queue, busy 60% of the time; requests arrive at random and take 10 ms of work on average. Its p99 is about 120 ms. An engineer sends every request to two servers at once and uses the first reply; the slower copy is not cancelled and still runs.

The interviewer asks: what does the p99 do over the next few minutes?

Question 4 of 6

A ticketing site's event page asks 8 venue shards in parallel and waits for all 8. A shard call's time is random with a long thin tail: half the calls finish within 7 ms, 99 in 100 within 46 ms, and the average is 10 ms (exponential). Shards are independent.

About what is the event page's p99?

Question 5 of 6

A video service's search page asks 50 index shards in parallel and waits for all. Calls take 5–15 ms, except during log compaction, when a shard's calls take 1 s. The operators run compaction on every shard at the same moments, so the shards are compacting together 1% of the time, and calls outside compaction are never slow.

To the nearest whole percent, what share of search pages take 1 s?

Question 6 of 6

A payments dashboard asks 30 merchant shards in parallel. The gateway sets one deadline, 300 ms from the start, and sends it with every call. A shard call runs three 20 ms stages; before each stage the shard checks the time left and, if none is left, answers at once that the deadline passed. On one load, 27 shards start at once, shard 19 first waits 200 ms in its queue, shard 12 waits 250 ms, and shard 7 waits 2 s.

How many shards' results does the dashboard show, and when?

Go deeper

Pro tips

When a page misses its latency target while every backend meets its own, list the calls the page waits for before tuning any one backend. When: a dashboard or page aggregates several services and only the page's percentile is bad (steps 1, 4). Verify:
```
tip the dashboard waits for 6 calls: a single widget is past 1 s on 1.0% of loads, the whole dashboard on 6.0%
```
Status: RAN.

Source

quote-check.mjs finds every passage below verbatim in its saved text. Valid as of 2026-10-15.

Jeffrey Dean and Luiz André Barroso, The Tail at Scale (Communications of the ACM 56(2), 2013), the authors' PDF: https://www.barroso.org/publications/TheTailAtScale.pdf (tail-at-scale.txt). The row's first source. Quoted: "If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second"; "naive implementations of this technique typically add unacceptable additional load"; "defer sending a secondary request until the first request has been outstanding for more than the 95th-percentile expected latency for this class of requests. This approach limits the additional load to approximately 5%"; "without synchronization, a few machines are always doing some background activity, pushing out the latency tail on all requests"; and "the user may be best served by being given slightly incomplete (“good-enough”) results in exchange for better end-to-end latency."
Google Research, The Tail at Scale (publication page): https://research.google/pubs/the-tail-at-scale/ (research-google-tail.txt). Quoted: "Temporary high latency episodes which are unimportant in moderate size systems may come to dominate overall service performance at large scale." Supports the headline.
Google SRE book, Addressing Cascading Failures: https://sre.google/sre-book/addressing-cascading-failures/ (sre-cascading.txt). The row's second source. Quoted: "The tree of RPCs emanating from an initial request will all have the same absolute deadline." and "you don’t get credit for late assignments with RPCs."

Steps 2, 4 and 5 are the arithmetic lib/tail.mjs runs, not quoted rules.

Checked against the tools and docs as of 2026-10-15.

Predict first