Skip to content

Distributed-systems basics

Distributed systems basics: partial failure, retries with jitter, circuit breakers, clocks, consensus, locks with fencing tokens, sagas and CQRS.

8 lessons

Partial failure and retries

  1. 1Partial failure and the ambiguous timeoutA timeout tells you when you stopped waiting, not what happened on the other side.
  2. 2Retries with exponential backoff and jitterA retry is more load on a service that is already failing. Spread retries in time, keep them to one layer, and cap their share.
  3. 3Circuit breakers and bulkheadsA breaker stops calling a failing dependency for a while, then tests it with one call; a bulkhead stops one slow dependency taking everyone's capacity.

Time, order and agreement

  1. 4Clocks and the order of eventsTwo clocks never agree exactly: let one place number the events, or carry the order inside the messages.
  2. 5Consensus and quorums (awareness)Count votes against the whole membership: only a majority decides, so a cut has at most one winning side.
  3. 6Distributed locks, leases and fencing tokensA lock in a shared store is a lease; only the resource itself can refuse a holder whose lease ran out.

Multi-step workflows

  1. 7Sagas: multi-step workflows without distributed transactionsNothing rolls back across services: every undo is a new step, run last-first, and allowed to run twice.
  2. 8CQRS and event sourcing (awareness)Keep the events, derive the rest: every read model can be rebuilt, and every event you store is kept for good.