Skip to content

Scale management

Scaling systems and teams: capacity math, tail latency, load testing, autoscaling, backpressure, data scaling, cost, ownership and release management.

27 lessons

Scaling systems

  1. 1Numbers every engineer should knowDistance sets a floor no server beats: count round trips, weigh them in powers of ten, and give each SSD figure its drive and year.
  2. 2Capacity from Little's law and the bottleneckThe slowest stage sets the pace. Everything let in beyond it only waits, and Little's law says for how long.
  3. 3Utilisation against latency, the queueing kneeRandom arrivals queue long before a server is full: time in the system is S ÷ (1 − ρ), so keep headroom and alert on saturation.
  4. 4Tail latency at scale: fan-out, hedging, deadlinesA page that waits for every call inherits the worst of them: fan-out multiplies the chance of meeting somebody's tail.
  5. 5Load testing: test types and open vs closed modelsPick the test by the question, and the model by how real users arrive.
  6. 6Autoscaling and cold startsDesired replicas and ready capacity are different states.
  7. 7Backpressure, admission control and priority sheddingWhen work arrives faster than it drains, push back early: queues sized to their callers' wait, admission by deadline, and shedding by criticality.
  8. 8The database scaling playbookSpend in order of cost: every step has a trigger you can measure, and sharding comes last because it breaks joins, transactions and uniqueness.
  9. 9Scaling the cache: hash slots, hash tags and moving slotsThe name picks the slot, the slot picks the primary: one call's keys share a slot, and clients follow MOVED and ASK.
  10. 10Scaling queue consumers: concurrency, partitions and orderingConsumers beyond the ordering units sit idle, and the busiest unit, not the average one, decides when the backlog is gone.
  11. 11CDN offload as a capacity leverAt scale the CDN is capacity: what melts is the origin's share, counted in requests and in bytes.
  12. 12Multi-region and data residencyWhere a tenant's writes land is a latency choice; where its copies live is a legal one.
  13. 13Cells and stamps: limiting blast radiusA cell turns "every tenant is down" into "one cell's tenants are down".
  14. 14Cost at scale: unit economics and FinOpsThe bill says how much; the unit cost says whether. Price each line per unit before cutting anything.
  15. 15Performance budgets at scaleA budget nobody's build can fail is a dashboard: it records the drift instead of stopping it.

Scaling codebases and teams

  1. 16Conway's law and team topologiesDraw the teams and you have drawn the system. Cut both along the work.
  2. 17Enforced module boundaries as the team growsA boundary is a rule CI runs: one public entry per module, and a check that fails forbidden imports and loops.
  3. 18Ownership: CODEOWNERS and service ownershipName one owner for every path and every service; for paths, the last matching line names it.
  4. 19Release management at scale: merge queues and release trainsTest each change on top of everything queued ahead of it before it lands; then ship what landed, versioned from the changes' own notes.
  5. 20Progressive delivery with automated analysisGrow the new build's share step by step, let a controller judge each step against written conditions, and abort on failure without waiting for anyone.
  6. 21Version skew: every release works with N−1Every release must read what its predecessor wrote, and write only what its predecessor can read, for as long as both can meet.
  7. 22Backfills and data migrations at scaleRewrite a live table as many small, repeatable, resumable batches, slowed by the replicas and proved before anything reads the result.
  8. 23Codebase-wide change: codemods and ratchetsWrite the edit as a program, ship it by owner, and keep a committed count that can only fall.
  9. 24Tech debt at scale: hotspots and a debt portfolioDebt interest gathers where complex code changes often: rank files by commits times complexity, fund the top, and judge it by that area's delivery.
  10. 25On-call load, toil and alert hygieneHalf the time for engineering. Page only what a person must act on now. Count the load and engineer the toil away.
  11. 26Platform engineering and paved roadsPave the road teams already want to walk, and keep it narrow.
  12. 27Design docs, RFCs and ADRs at scaleWrite it when the choice is hard. Show the trade-offs. Close it on a date, with one decider, and keep the call as an ADR.