Partial failure and the ambiguous timeout

A timeout tells you when you stopped waiting, not what happened on the other side.

Before you read: predict

A shop's cart page reserves the last desk lamp for cart c-4 by calling the stock service, with a 3 s timeout. No reply arrives within 3 s, so the call times out. A moment later the cart reads the stock service's reservations for c-4 and finds none.

Is the lamp reserved for c-4?

The 30-second version

A remote call can fail in three places: the request is lost, the server finishes too late, or the reply is lost. The caller sees the same silence, so a timeout means "unknown", not "failed". Give each call a deadline from the caller's budget; pass it on. Settle the outcome from the server's record once late work cannot land.

A timeout tells you when you stopped waiting, not what happened on the other side.

How it works

Step 1 of 14

A remote call is three separate events: the request crosses the network, the server does the work, the reply crosses back.

In the kitchen

the order bird out, the reply bird back → request and reply (step 1)

Analogy: An island harbourmaster's pigeon post

The harbourmaster orders fuel from the mainland depot by pigeon and waits for the return bird. The fishing fleet sails at dawn.

At the harbour	In the system
the order bird out, the reply bird back	request and reply (step 1)
a hawk takes the order bird	request lost (step 2)
the depot loads the boat after his cut-off hour	work done late (step 3)
the depot loads and sends the reply bird; a hawk takes it	reply lost (step 4)
at the cut-off, an empty loft: he cannot tell which happened	the timeout means "unknown" (step 5)
he writes "fuel unconfirmed" and asks the morning boat what the depot did	learn the outcome from the server's record (step 6)
the bird comes straight back, order still tied on: the depot's loft was boarded up	a refused connection (step 7)
his cut-off is dawn minus the hours to fill the tanks	the deadline from the caller's budget (step 10)
the slip says "wanted by midnight", so the depot checks the hour before loading	deadline propagation (steps 12–13)

In real terms: record a timed-out call as unconfirmed, then settle it from the server's own state.

Breaks down when: you picture him rowing across to look. A caller often has no second channel: it can only ask the same server again, over the same network.

Contrasting case: a reply bird that arrives before the cut-off. Nothing is unknown, and no deadline decides anything.

Examples, run for real

4.1 ExamHall: "Pause failed" on paused candidates

EXH-1710: a proctor pressed Pause; the proctor API called the session service with a 2 s deadline and showed every timeout as "Pause failed". The script runs the same call once per fault:

4.1 proctor p-12 pauses candidate c-311; the proctor API calls the session service with a 2000 ms deadline
  run request-lost  proctor API: no reply by 2000 ms, TimeoutError · session service saw the request: no  · committed: no
  run done-late     proctor API: no reply by 2000 ms, TimeoutError · session service saw the request: yes · committed: yes (at 2300 ms)
  run reply-lost    proctor API: no reply by 2000 ms, TimeoutError · session service saw the request: yes · committed: yes (at 0 ms)
  as shipped   every timeout shown as a failure    tile: "Pause failed", "Pause failed", "Pause failed" · wrong in 2 of 3
  MR !1910     timeout shown as "Pause unconfirmed", then GET /sessions/c-311 at 5000 ms: "running", "paused", "paused" · wrong in 0 of 3

The proctor API's view is identical; the session's state is not (steps 2–5). MR !1910 shows "unconfirmed" and reads the session after the late pause has landed (step 6).

4.2 ExamHall: enrichment for nobody

EXH-1711: the live feed enriches each alert through risk scoring and a three-stage image check. Every hop used its own 10 s timeout, although the console waits 2 s. MR !1911 propagates the console's deadline.

4.2 proctor console enriches alert al-77 within 2000 ms; the image check runs 3 stages of 700 ms each
  as shipped (each hop its own 10000 ms timeout)
    hops: proctor API own 10000 ms -> risk own 10000 ms -> image-check own 10000 ms
    console: no reply by 2000 ms, TimeoutError · image check: stage 1 400-1100, stage 2 1100-1800, stage 3 1800-2500 · stages finished after the console stopped waiting: 1
  MR !1911 (deadline propagated, checked before each stage)
    hops: proctor API 2000 ms left -> risk 2000 ms left -> image-check 1600 ms left
    console: reply 504 at 1800 ms · image check: stage 1 400-1100, stage 2 1100-1800, stage 3 skipped at 1800 · stages finished after the console stopped waiting: 0

With 200 ms left, stage 3 is skipped, and the console gets an answer in time instead of silence (steps 12–13).

4.3 Proof domain, catalog: a call with no deadline

4.3 20 product pages, one every 100 ms; the reviews service accepts connections and never answers; pool of 10 sockets
  no deadline         pages loaded within 1 s: 0/20 · calls holding a socket at 3000 ms: 10 · waiting for a socket: 10
  300 ms deadline     pages loaded within 1 s: 20/20 (without reviews) · calls holding a socket at 3000 ms: 0 · waiting for a socket: 0

A silent service returns no error, so nothing ends the call; ten calls hold every socket and the rest queue behind them (step 8).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 5

A product page must be sent within 1,000 ms of its request. At 50 ms it calls the recommendations service; after that call it needs 150 ms more to render, with or without recommendations. Of that service's last 1,000 calls, 990 were answered, with a median of 87 ms and a p99 of 492 ms; 10 hung.

Which timeout should the page's call get?

Question 2 of 5

An admin tool sets new prices through the catalog service with a 2 s timeout. Last night three calls timed out. The catalog's own log shows: sku-1, no entry; sku-2, received at once and price applied at 2.4 s, its reply written to a closed connection; sku-3, received, price applied and answered 200 at once. The team proposes a 30 s timeout.

Had the same three calls run with it, which would the tool have logged as "updated"?

Question 3 of 5

A product page's deadline is 1,000 ms after it starts. Search spends 300 ms, then calls the pricing service, forwarding that deadline:

Four such requests wait in pricing's queue for 200, 650, 800 and 1,100 ms before a worker takes each.

How many of the four prices does pricing finish after the page has stopped waiting?

// pricing service; x-deadline is the page's deadline, forwarded unchanged by searchhttp.createServer((req, res) => {  if (clock.now() >= Number(req.headers['x-deadline'])) return res.writeHead(504).end();  queue.push(() => res.end(computePrice(req)));   // a worker runs it later; computing takes 100 ms});

Question 4 of 5

A product page calls the reviews service with Node's socket.setTimeout(2000) as its only timeout, and renders once that call ends, with a "reviews unavailable" note if the call failed. Today the reviews service sends its reply one byte every 400 ms, on and on, and product pages hang.

Which change shows every page, with its reviews or the note, within 1 s?

Question 5 of 5

A cart asks the stock service to reserve mug-5. The call can fail in four ways.

After which one can the cart tell the shopper for certain that mug-5 is not reserved?

Go deeper

Pro tips

Test timeout handling with a proxy that drops bytes, not with a slow mock. A delaying mock covers only the late finish; the lost request and the lost reply need something that swallows bytes in one direction. When: writing tests for any client that calls another service (steps 2–4).
Log the time left at every hop. Carry the remaining budget in each outgoing call's header and log it on arrival, so a trace shows which hop spent it. When: after adding deadline propagation (step 12).

Source

Jacob Gabrielson, "Challenges with distributed systems", Amazon Builders' Library (PDF): https://d1.awsstatic.com/builderslibrary/pdfs/challenges-with-distributed-systems.pdf · fetched 2026-09-29. The eight request/reply steps and their failures: "POST REQUEST fails", "DELIVER REQUEST fails: NETWORK successfully delivers MESSAGE to SERVER, but SERVER crashes right after it receives MESSAGE", "POST REPLY fails", "DELIVER REPLY fails"; "If a reply is never received, time out. In this step, timing out means that the result of the request is UNKNOWN. It may or may not have happened." A refused connection is the exception: "the client knows, deterministically, that the request could not possibly have been received by the server machine." Summary: "The result of any network operation can be UNKNOWN, in which case the request may have succeeded, failed, or received but not processed" (steps 1–5, 7). Valid as of 2026-09-29.
"Timeouts, retries, and backoff with jitter", Amazon Builders' Library, now on AWS Builder Center (author shown as Marc; page dated Jun 12, 2026): https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter · fetched 2026-09-29 (the old aws.amazon.com/builders-library URL redirects there). "A timeout or failure doesn't necessarily mean that side effects haven't happened"; "A best practice in Amazon is to set a timeout on any remote call"; "This includes both a connection timeout and a request timeout"; "we choose an acceptable rate of false timeouts (such as 0.1%). Then, we look at the corresponding latency percentile on the downstream service"; "There are also implementations where the timeout doesn't cover all remote calls, like DNS or TLS handshakes"; SO_RCVTIMEO is "unsuitable as an end-to-end socket timeout" (steps 5, 8, 9, 11). Its retry and jitter sections belong to ds-retries-backoff-jitter. Valid as of 2026-09-29.
Google SRE book, ch. 22 "Addressing Cascading Failures": https://sre.google/sre-book/addressing-cascading-failures/ · fetched 2026-09-29. "It's usually wise to set a deadline. Setting either no deadline or an extremely high deadline may cause short-term problems that have long since passed to continue to consume server resources"; "the server should check the deadline left at each stage before attempting to perform any more work on the request"; "Rather than inventing a deadline when sending RPCs to backends, servers should employ deadline propagation"; "The tree of RPCs emanating from an initial request will all have the same absolute deadline"; "You may want to reduce the outgoing deadline a bit (e.g., a few hundred milliseconds) to account for network transit times"; "Having deadlines several orders of magnitude longer than the mean request latency is usually bad". Valid as of 2026-09-29.
Peter Deutsch, "The Eight Fallacies of Distributed Computing", as published on James Gosling's site: https://nighthacks.com/jag/res/Fallacies.html · fetched 2026-09-29. "Essentially everyone, when they first build a distributed application, makes the following eight assumptions"; "1. The network is reliable", "2. Latency is zero" (step 14). The list's history (seven fallacies in 1994, the eighth added by Gosling around 1997) is only in secondary sources and is not claimed here. Valid as of 2026-09-29.
Node.js v24 docs, net, socket.setTimeout(timeout[, callback]): https://nodejs.org/docs/latest-v24.x/api/net.html · fetched 2026-09-29. "Sets the socket to timeout after timeout milliseconds of inactivity on the socket. By default net.Socket do not have a timeout." "When an idle timeout is triggered the socket will receive a 'timeout' event but the connection will not be severed". Valid as of 2026-09-29.
Model, not observed: P1's latency sample is synthetic (a seeded log-normal draw, median 87 ms, p99 492 ms, 10 hung calls) replayed on a virtual clock; no real service was measured. D1's idle and connect timers are the script's own, on the virtual clock, reset on each received byte as the Node docs above describe socket.setTimeout. Before any key is graded, a precondition runs Node's real socket.setTimeout(500) in real time against a real server trickling a byte every 100 ms (0 'timeout' events in 3 s) and a silent one (at least 1 event, connection still open); it counts events with a 5× margin and exits 3 if either count is off. The proxy's "lost request" swallows bytes on one connection; real packet loss would first be retried by TCP, so the model stands for a loss TCP cannot hide (a crashed router or middlebox, a dropped connection).

Checked against the tools and docs as of 2026-09-29.

Predict first