Deployment strategies and rollback

Rolling swaps pods in bounded batches, blue-green flips all traffic, a canary is judged against its control; rollback restores code, not data.

Before you read: predict

The shop's checkout service runs on 20 identical pods behind a load balancer that spreads checkouts evenly over them. At 14:00 build 8.2 replaces build 8.1 on one pod; the other 19 keep 8.1. Build 8.2 has a bug: every checkout that carries a coupon code fails with a 500. About one checkout in five carries a coupon. On any build, about 1 checkout in 200 fails with a 504 when the payment provider times out. 200 checkouts arrive each minute. At 14:15 the canary gate runs once:

What does the gate do?

const { requests, errors } = metrics.checkout({ since: '14:00' });   // counted at the load balancerif (errors / requests > 0.02) rollBack('8.2'); else promote('8.2');  // promote: put 8.2 on all 20 pods

The 30-second version

A deploy strategy sets how traffic reaches a new build. A rolling update replaces pods in batches bounded by maxSurge and maxUnavailable, so both builds serve at once. Blue-green flips all traffic between two full environments. A canary sends a small share; judge it against the control, since a fleet-wide rate dilutes it. Rollback restores code, never data.

Rolling swaps pods in bounded batches, blue-green flips all traffic, a canary is judged against its control; rollback restores code, not data.

How it works

Step 1 of 11

Rolling update. New pods start and old ones stop in batches; until the last batch, both builds serve requests, often one user's in turn.

In the kitchen

two of ten stalls refitted at a time, never more than two out of use → a rolling update inside its bounds (steps 1–2)
for a week, some cows milked on old machines and some on new → both builds serving at once (step 1)

Analogy: A dairy farm changing its milking machines

The herd is milked twice a day, every day, so the machines have to change while milking goes on.

On the farm	In the deploy
two of ten stalls refitted at a time, never more than two out of use	a rolling update inside its bounds (steps 1–2)
for a week, some cows milked on old machines and some on new	both builds serving at once (step 1)
a second parlour fitted out beside the old one, and the herd sent through its gate one morning	blue-green (step 4)
the old parlour kept ready, so the gate can swing back	the instant way back (step 5)
one stall fitted first, its milk tested against the other nine stalls at the same milkings	a canary judged against its control (steps 6–7)
that stall's milk compared with last month's, when the feed was different	a before-and-after comparison (step 8)
milk the faulty machine already sent off in the tanker	what a rollback leaves behind (step 10)

In real terms: a rolling update mixes builds, blue-green swaps environments, a canary splits traffic for a measured comparison, and any way back changes code only.

Breaks down when: a cow stays in one stall for a whole milking, while a user's next request may land on either build. A tanker can be traced and its milk poured away; a sent email cannot be unsent.

Contrasting case: a farm that stops milking for a morning to swap every machine has one kind of machine at a time, and a morning of cows waiting. That is Recreate.

Examples, run for real

4.1 ExamHall EXH-2090: a rolling update that reached every candidate

EXH-2090: the candidate API's build 7.4 failed 1 autosave in 10, and it went out as a rolling update to all 20 API pods. MR !2390 keeps deploys out of exam windows and, outside them, runs 7.4 on 1 pod of 20 for 15 minutes, gated on that pod's autosave error rate against the other pods' over the same minutes.

4.1 before: rolling update of all 20 pods   7.4 behind every autosave from minute 2 · rollout undo run then: a rolling update back, done at minute 6 · failed autosaves 70
4.1 after (MR !2390): 1 of 20 pods, 15 min   7.4 behind 5% of autosaves · gate at minute 15: 7.4 10.0% vs 7.3 0.0% -> roll back · failed autosaves 15

The rolling update put 7.4 behind every autosave within two minutes, and its undo was another rolling update (steps 1, 11). The canary exposed one pod's share for 15 minutes, and the gate saw the bug at full strength in that pod's own rate (steps 7, 9).

4.2 Proof domain, checkout: both builds serve during a rolling update

Four checkout pods move from build 2.0 to 2.1 with maxSurge: 1 and maxUnavailable: 0; a new pod is ready one minute after it starts, and 20 checkouts arrive each minute.

4.2 minute 0  pods 2.0 x4 + 2.1 x1 · checkouts served by 2.0: 20 · by 2.1:  0 · failed 0
4.2 minute 1  pods 2.0 x3 + 2.1 x2 · checkouts served by 2.0: 15 · by 2.1:  5 · failed 0
4.2 minute 2  pods 2.0 x2 + 2.1 x3 · checkouts served by 2.0: 10 · by 2.1: 10 · failed 0
4.2 minute 3  pods 2.0 x1 + 2.1 x4 · checkouts served by 2.0:  5 · by 2.1: 15 · failed 0
4.2 minute 4  pods 2.0 x0 + 2.1 x4 · checkouts served by 2.0:  0 · by 2.1: 20 · failed 0
4.2 minutes in which both builds served checkouts: 3 · most pods at once 5 · fewest available 4

For three minutes, checkouts went to both builds; the pod count never passed five, and four pods stayed available throughout (steps 1–2).

4.3 Proof domain, checkout: a blue-green switch and its way back

Blue runs build 2.1 on four pods; green runs 2.2, which fails 1 checkout in 5, on four more. The router sends everything to green at minute 0, and back to blue at minute 3.

4.3 minute 0  router -> green · checkouts served by 2.2 · failed  4 · pods running 8
4.3 minute 1  router -> green · checkouts served by 2.2 · failed  4 · pods running 8
4.3 minute 2  router -> green · checkouts served by 2.2 · failed  4 · pods running 8
4.3 minute 3  router -> blue  · checkouts served by 2.1 · failed  0 · pods running 8
4.3 minute 4  router -> blue  · checkouts served by 2.1 · failed  0 · pods running 8
4.3 minute 5  router -> blue  · checkouts served by 2.1 · failed  0 · pods running 8

All traffic moved at once, so every checkout met 2.2 for three minutes, with eight pods running throughout. Blue was still running, so the switch back took effect in the same minute (steps 4–5).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 5

A catalog API's Deployment is updated from build 3.4 to 3.5:

spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

A new pod passes its readiness probe 2 minutes after it starts, and none fails.

During the update, what is the largest number of catalog pods that exist at one moment, and the smallest number that are available?

Question 2 of 5

The checkout service runs eight pods of build 3.9 behind a load balancer that sends each request to any ready pod in its target set. Build 4.0 stores the basket in a new cookie format: 4.0 reads both formats, but 3.9 cannot read 4.0's and starts an empty basket. A new pod is ready 1 minute after it starts. 40 shoppers each send a request every minute.

Which release plan moves the service to 4.0 with no shopper's basket emptied and no request answered with a 503?

Question 3 of 5

A shop's order service runs build 6.1. Build 6.2 changes how an order's delivery address is saved:

Yesterday 6.2 ran as a canary on 1 of the service's 10 pods, with a 25% router weight, from 09:00 to 09:30, while 20 orders were placed each minute. At 09:30 a latency alarm fired and the canary pod went back to 6.1; 1,400 more orders were placed through the rest of the day. This morning each of yesterday's customers opens their order page once.

How many of those page loads fail?

// build 6.1, order pagehtml += `<p>Deliver to: ${order.address.trim()}</p>`;// build 6.2, saving an orderorders.insert({ ...order, address: { line1: form.line1, city: form.city, postcode: form.postcode } });

Question 4 of 5

A video service releases each build blue-green. At 20:00 the router sends every stream start to green, the new build; at 20:15 a gate compares green's failure rate for 20:00–20:15 with blue's for 19:45–20:00, and rolls back if green's is more than 5 points higher. Normally 1 stream start in 100 fails. Last Tuesday's build had no defect, but a cup final kicked off at 20:00 and the licence service failed 1 call in 10 until 22:00; the gate rolled back. Last Thursday's build failed 1 stream start in 10 itself, on a quiet evening, and the gate rolled back. 50 stream starts arrive each minute.

Which change to the gate would have promoted Tuesday's build and still rolled back Thursday's?

Question 5 of 5

A ticket shop released build 5.1 with this script. Each line finishes before the next starts, and a new pod is ready 3 minutes after it starts.

Build 5.1 failed 1 ticket purchase in 4. The alarm fired 4 minutes after line 3, the on-call ran lines 6 and 7, and purchases kept failing for 3 more minutes.

Which line made the way back take minutes?

1  await startPods('green', '5.1', 6);2  await waitUntilReady('green');3  await router.sendAllTo('green');4  await scalePods('blue', 0);        // blue ran 5.05  // on the error alarm:6  await router.sendAllTo('blue');7  await scalePods('blue', 6);

Go deeper

Pro tips

Set minReadySeconds above the time a bad build takes to crash after its readiness probe first passes: such a pod never counts as available, so the rolling update stops with the old pods serving. When: a build that can pass its probe and then fall over on real traffic (steps 2–3). Verify (RAN, tips.mjs, on the controller model with 10 real pods; the new build crashes 1 minute after it first reports ready):
```
tip1 minReadySeconds 0    old 9.0 pods left after 12 minutes: 3 · fewest ready pods in a minute: 3 of 10 · requests answered 503: 0
tip1 minReadySeconds 180  old 9.0 pods left after 12 minutes: 8 · fewest ready pods in a minute: 8 of 10 · requests answered 503: 0
```

Source

The Kubernetes docs can change, so the rollout rules are valid as of 2026-10-01.

Google SRE Workbook, Canarying Releases (Warner and Davidovič), the curriculum row's first source: https://sre.google/workbook/canarying-releases/ (sre-canarying.txt). "We define canarying as a partial and time-limited deployment of a change in a service and its evaluation"; "If we instead use a canary population of 5%, we serve 20% errors for 5% of traffic, resulting in a 1% overall error rate"; "Because time is one of the biggest sources of change in observed metrics, it is difficult to assess degradation of performance with before/after evaluation"; blue/green "rollback is a trivial reversal of the router change. One downside is that this setup uses twice as many resources".
Kubernetes docs, Deployments, the curriculum row's second source: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ (k8s-deployment.txt). maxUnavailable: "The absolute number is calculated from percentage by rounding down"; maxSurge: "The absolute number is calculated from the percentage by rounding up. The default value is 25%"; "The Deployment controller stops the bad rollout automatically, and stops scaling up the new ReplicaSet"; "when you roll back to an earlier revision, only the Deployment's Pod template part is rolled back"; "All existing Pods are killed before new ones are created when .spec.strategy.type==Recreate"; minReadySeconds is "the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available". The controller model in lib/fleet.mjs implements these rules.
Martin Fowler, BlueGreenDeployment, 1 March 2010: https://martinfowler.com/bliki/BlueGreenDeployment.html (fowler-bluegreen.txt). "you switch the router so that all incoming requests go to the green environment"; "if anything goes wrong you switch the router back to your blue environment".
Danilo Sato, CanaryRelease, martinfowler.com, 25 June 2014: https://martinfowler.com/bliki/CanaryRelease.html (fowler-canary.txt). "slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure". Supports step 6.

Read first: CI/CD pipelines, build once and promote everywhere

Checked against the tools and docs as of 2026-10-01.

Predict first