Availability math, single points of failure and failover

Multiply what a path needs in series, count copies only when they fail apart, and price every failover in detection, promotion and re-routing.

Before you read: predict

ExamHall's exam-start request calls the login API, then the session store, then the exam database, and needs an answer from each. Last quarter the login API and the session store were each up 99.95% of the time. ExamHall promises schools that exam start will be up 99.8% of the time. The three fail independently of each other.

What is the lowest availability the exam database can have while that promise still holds, to two decimal places?

The 30-second version

A path that needs every component is up only when all of them are, so independent availabilities multiply and the path sits below its weakest part. Redundant copies fail only together. A single point of failure is a part whose loss alone stops the path. Failover takes detection, promotion and re-routing; active-active survivors need headroom.

Multiply what a path needs in series, count copies only when they fail apart, and price every failover in detection, promotion and re-routing.

How it works

Step 1 of 13

Availability is the share of time, or of requests, that a component works. Turning a target into minutes per window belongs to se-requirements-nfrs.

Analogy: A shop's monitored alarm

A break-in reaches the alarm company's receiving centre through a chain: sensor, panel, communicator, phone line. The centre then rings the shop's keyholders in turn.

At the shop	In a system
the call-out happens only if the sensor, panel, communicator and line all work	a path in series (steps 2–3)
a second path over the mobile network gets the signal through if either path works	copies in parallel (step 4)
both paths start at the one panel, on its one battery	a shared dependency behind a pair (steps 5, 8)
the second keyholder is rung only after the first has not answered for ten minutes, then drives in	active-passive: every phase is downtime (step 9)
two patrol officers each cover half the shops, and each must cover all of them when one is off sick	active-active headroom (step 12)
the new keypad code, written only in the first keyholder's notebook	writes the standby never received (step 11)

Breaks down when: keyholders know about each other, and database nodes do not. An old primary that comes back after a failover does not know it was replaced, so a guard has to stop it taking writes (step 13; db-replication-read-replicas runs that case).

Contrasting case: a cut phone line outside the shop leaves the mobile path working, but a flood that soaks the panel stops both paths at once. A copy protects only against the failures it does not share.

In real terms: a design's availability follows from which parts it needs together, which parts have copies that fail apart, and what switching to a copy takes.

Examples, run for real

4.1 ExamHall EXH-2170: every service met its target, the path did not

ExamHall promised schools 99.9% for exam start because login, the session store, the paper service and the exam database each met 99.9% on its own. Exam start needs all four.

4.1 exam start needs 4 services, each 99.9%: path 99.60% (16 states enumerated; the product gives 99.60%)
4.1 expected downtime per 30-day month: each service 43.2 min · the path 172.5 min

The path sits below every service it needs (steps 2–3), and its expected downtime is about four times one service's. MR !2470 publishes the path's product as the promise, and gives each one-node service on it a copy with automatic failover.

4.2 Checkout: what a copy buys, and what a shared host takes back

The checkout's app instances are each up 99.5% of the time; either one can serve all of checkout.

4.2 app instances at 99.5% each: one 99.5% · two 99.9975% · three 99.999988%
4.2 two instances on one host (host 99.9%): 99.8975% · the host alone caps them at 99.9%

Each copy multiplies the tier's unavailability by 0.005 (step 4). Put both copies on one host, and the host stands in series with the pair: its 0.1% swamps the pair's 0.0025% (steps 5, 8).

4.3 ExamHall: how many failovers fit in a month

Exam windows target 99.95% over a 30-day month (minutes from se-requirements-nfrs' rule).

4.3 budget at 99.95% per 30-day month: 21.6 min
4.3 automatic failover, 90 s end to end: 14 fit · promotion by hand from a runbook, 12 min: 1 fits

A short, automated failover is what lets the target survive a failure at all (steps 9 and 13).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 5

The checkout runs through one load balancer, then the app tier, then one payment-gateway connector, and needs all three. The app tier is two instances in different zones: the balancer stops sending to a dead one, and either instance alone can carry all of checkout. Each instance is up 99% of the time, the balancer 99.99% and the connector 99.95%. Failures are independent, and moving traffic between the instances costs nothing.

What share of the time is checkout up, to two decimal places?

Question 2 of 5

A shop's checkout is built like this, and every checkout needs all of the following:

one HAProxy VM in front of two app instances; either app instance can serve all of checkout, and HAProxy stops sending to a dead one;
a tax table, which each app instance reads on every checkout from the NFS share that both instances mount;
the cart, in Redis: a primary, plus a replica that copies every write automatically and feeds the nightly sales report; the app's configuration names only the primary;
the order, in Postgres: a primary and a standby, with an automatic failover tool that promotes the standby and moves the DNS name, tested every month.

Of HAProxy, the NFS share, the Redis primary and the Postgres primary, which one can fail on its own without stopping checkout until a person steps in?

Question 3 of 5

The payments API runs active-active on four nodes behind a balancer that spreads requests evenly over the healthy nodes. Each node can handle 400 requests a second and rejects the rest with a 503. If a node fails, the balancer stops sending to it.

What is the highest evening peak, in requests a second, at which the API rejects nothing even after one node has failed?

Question 4 of 5

A ticket shop's database runs active-passive. A monitor checks the primary every 10 seconds, on the minute and at :10, :20 and so on, and declares it dead after 3 failed checks in a row. Promoting the standby then takes 20 seconds, after which the DNS name points at the standby. Each app server keeps a DNS answer for 60 seconds from the moment it looks the name up, and looks it up again only when that answer expires; all of them last looked it up at 13:59:45. The primary's host dies at 14:00:05; the check at 14:00:00 passed. Orders arrive every second.

At what time does the first order succeed again?

Question 5 of 5

ExamHall's answer store runs on Postgres. The primary, the failover tool's VM and two of the four app servers are in zone A; the standby and the other two app servers are in zone B, behind a load balancer that spans both zones. The primary streams its changes to the standby asynchronously. The failover tool checks the primary every 10 seconds and, after 3 failed checks, promotes the standby, which takes 20 seconds, and moves the DNS name. The app servers cache the name's answer for 30 seconds. The standby serves no queries. On Tuesday zone A lost power from 10:00 to 10:40, and saving answers failed for all 40 minutes.

Which one change would have brought answer saving back within two minutes?

Go deeper

Pro tips

Before you quote one availability for a design, list the services a request needs and multiply their figures; quote the product, never one service's own figure. When: a review, an SLA or an interview answer asks how available a path is (steps 2–3). Verify: tip three 99.9% services in series: 99.70% · 129.4 min down per 30-day month (2 h 9 min) Status: RAN.

Source

Vendor behaviour can change: the RDS facts below are as of 2026-10-01. "The Calculus of Service Availability" (ACM Queue) answered 403 that day and is not used. Valid as of 2026-10-01.

Google, Site Reliability Engineering, ch. 3 Embracing Risk: https://sre.google/sre-book/embracing-risk/ (sre-embracing-risk.txt). The curriculum row's first source. Quoted: "Each additional nine corresponds to an order of magnitude improvement toward 100% availability" and "the user experience is dominated by less reliable components like the cellular network or the device they are working with". Supports steps 1 and 3.
AWS whitepaper Availability and Beyond, Availability with dependencies: https://docs.aws.amazon.com/whitepapers/latest/availability-and-beyond-improving-resilience/availability-with-dependencies.html (aws-availability-dependencies.txt). Quoted: the theoretical maximum availability "is the product of the availability of all of the dependencies … because each one must be operational", and "computing a maximum theoretical availability is only likely to produce a rough order of magnitude calculation".
AWS whitepaper Availability and Beyond, Availability with redundancy: https://docs.aws.amazon.com/whitepapers/latest/availability-and-beyond-improving-resilience/availability-with-redundancy.html (aws-availability-redundancy.txt). Quoted: "For the whole system to be down, both subsystems must be down at the same time"; a workload using spares "must know how to identify dependency failure, weight work away from it to a healthy resource, and manage overall capacity of the workload"; and "an entire AZ worth of Amazon EC2 instances can fail together".
PostgreSQL documentation, 26.3 Failover (current, version 18 on 2026-10-01): https://www.postgresql.org/docs/current/warm-standby-failover.html (pg-failover.txt). The curriculum row's third source. Quoted: "PostgreSQL does not provide the system software required to identify a failure on the primary and notify the standby database server", and STONITH is "necessary to avoid situations where both systems think they are the primary". Supports steps 9 and 13 and the failover model's detect-then-promote order.
Amazon RDS User Guide, Multi-AZ DB instance deployments: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html (rds-multiaz-single.txt), and Configuring and managing a Multi-AZ deployment: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html (rds-multiaz.txt). The curriculum row's second source. Quoted, as of 2026-10-01: a Multi-AZ DB instance deployment "has one standby DB instance that provides failover support, but doesn't serve read traffic", kept as "a synchronous standby replica in a different Availability Zone". Supports steps 9 and 11 (an active-passive pair).
Amazon RDS User Guide, Failing over a Multi-AZ DB instance: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.Failover.html (rds-failover.txt). Quoted, as of 2026-10-01: "Failover times are typically 60–120 seconds", and the failover "automatically changes the Domain Name System (DNS) record of the DB instance to point to the standby", while a JVM "caches the IP address for a specified period of time, known as the time-to-live (TTL)".

Checked against the tools and docs as of 2026-10-01.

Predict first