Progressive delivery with automated analysis

Grow the new build's share step by step, let a controller judge each step against written conditions, and abort on failure without waiting for anyone.

Before you read: predict

ExamHall releases a grading-rules change to its API at 02:00 with an Argo Rollouts canary, while night sittings run and nobody watches. The steps are setWeight: 5, pause: {duration: 15m}, setWeight: 50, pause: {duration: 30m}, and after the last step the new build takes every request. A background analysis runs from the first step: every 5 minutes, starting 02:05, it measures the new build's autosaves over the last 5 minutes, with successCondition: error rate below 1% and p95 below 0.8 s, and no failureLimit set. The new build's p95 stays at 0.4 s. Its error rate is 0.2% until the first sitting ends at 02:20, and 3% from then on.

When does the new build stop taking autosaves?

The 30-second version

Progressive delivery gives a new build growing slices of traffic (pods, users or rings of tenants), each held for a bake time. A controller measures the slice's metrics on a schedule and checks every measurement against conditions written beforehand. One failed measurement aborts the release and sends all traffic back to the old build, with nobody watching.

Grow the new build's share step by step, let a controller judge each step against written conditions, and abort on failure without waiting for anyone.

How it works

Step 1 of 13

Growing slices. The new build's share grows step by step, each share held for a bake time, so a fault meets few users first.

In the kitchen

the new schedule on one green first, then the front nine, then all eighteen, a week each → growing slices, each held for a bake time (steps 1–2)

Analogy: A golf club's greenkeepers moving the greens to a new night watering schedule

The sprinklers run after midnight, when no greenkeeper is on the course.

On the course	In the release
the new schedule on one green first, then the front nine, then all eighteen, a week each	growing slices, each held for a bake time (steps 1–2)
every few minutes the irrigation computer reads each green's soil-moisture probes and checks them against levels set before the trial	an analysis measuring on a schedule against written conditions (steps 3, 5)
a green reading drier than the limit goes back to the old schedule at 3 a.m. by itself	a failed measurement aborting at once (step 6)
a reading between the target and the limit	a measurement that meets neither condition (step 5)
every green drawing from the same pond and pump	canary and control sharing what they depend on (step 10)

In real terms: before the release, write what each step measures and what counts as a failure, and let the controller act on it at any hour.

Breaks down when: a green put back on the old schedule recovers by the next night, while requests the new build served, and rows it wrote, stay served and written after an abort.

Contrasting case: a greenkeeper walking the trial green at dawn and judging it by eye. That is a manual canary: it protects only on the mornings someone walks the course.

Examples, run for real

4.1 ExamHall EXH-4350: a night release that only its timers moved

EXH-4350: last term a grading-rules change went out at 02:00 as 5% of API pods for 15 minutes, then 50% for 30 minutes, then every pod, moved on by the steps' timers alone; nobody was awake. Once the first night sitting ended at 02:20, each autosave on the new build waited on mark recomputation. MR !4650 adds a background analysis from the first step: every 5 minutes, autosave errors below 1% and p95 below 0.8 s. Night traffic is 1,200 autosaves a minute.

4.1 EXH-4350 timers only   new build p95 1.40 s from 02:20 · 50% until 02:45, then every pod · rolled back at 07:30 · slow autosaves 357,000
4.1 MR !4650 with analysis  02:25 p95 1.40 s Failed -> abort, canary weight 0 at 02:25 · slow autosaves 3,000

The timers moved the build on whatever it did (step 1). The analysis failed at its first measurement after the slowdown, and the weight went to zero in the same minute (steps 4, 6).

4.2 ExamHall's rings, never during an exam window

MR !4650 also splits the institutions into rings: ExamHall's own test exams, then two pilot universities, then everyone. Each ring bakes for 24 hours under the same analysis, and starts only when none of its institutions has an exam window open or opening within 6 hours.

4.2 ring 1 internal test exams     institutions  1 · starts Mon 09:00 · analysis passed Tue 09:00
4.2 ring 2 two pilot universities  institutions  2 · starts Tue 16:00 · analysis passed Wed 16:00
4.2 ring 3 every institution       institutions 40 · starts Wed 17:00 · analysis passed Thu 17:00

Ring 2 waited for both pilots' Tuesday windows to close, and ring 3 for Wednesday's last one (step 13).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

A chat app releases a new message-search build with an Argo Rollouts canary at 23:00 on Friday. The steps are setWeight: 20, pause: {duration: 10m}, an inline analysis step, setWeight: 60, pause: {duration: 10m}, and after the last step the new build takes every search. The inline analysis takes one measurement of the new build's search p95, with successCondition: p95 at or below 0.3 s and failureCondition: p95 above 0.6 s. A background analysis also runs from the first step: every 10 minutes, starting 23:10, it measures the new build's search error rate over the last 10 minutes, with successCondition below 1%. The new build's p95 is 0.45 s all night. Its error rate is 0.2%, except from 02:00 to 04:00, while the nightly index rebuild runs, when it is 3%. The on-call engineer checks the rollout at 08:00: a paused rollout is resumed, an aborted one is left as it is.

Which timeline does the rollout follow?

Question 2 of 7

A chat app's send service releases a new build with an Argo Rollouts canary. At 01:05 the rollout sits at its 50% step, pause: {duration: 30m}, reached at 01:00. The on-call engineer sees the send p95 of both builds at 0.9 s since 01:00, against 0.12 s before. The background analysis compares the two builds over the same 5 minutes and passes while the new build's p95 is at most 1.2 times the old build's; it has passed every measurement. Two explanations fit: the message-history query in the new build loads the database both builds share, or the database slowed on its own at 01:00.

Which check, started now, tells them apart?

Question 3 of 7

In a mock interview you said a chat app's notification service should release behind a canary. The interviewer follows up: "At 03:00 the service sends 60 notifications a minute, and a canary gets 5% of them. Normally 1 send in 1,000 fails. Your setup must abort a build that fails 1 send in 50 at least 9 times in 10, and promote a healthy build at least 9 times in 10. What do you set up?" Three candidates answered:

Whose answer survives the follow-up?

Question 4 of 7

A chat app's presence service (who is online) releases a new build with Argo Rollouts steps setWeight: 10, pause: {duration: 5m}, an inline analysis step, setWeight: 50, pause: {duration: 30m}; after the last step the new build takes every update. The analysis takes one measurement, as soon as its step is reached, of the new build's presence-update error rate over the last 5 minutes, with successCondition below 1%. The service handles 6,000 presence updates a minute. The new build fails 0.1% of updates while its pods carry up to 2,000 a minute, and 6% once they carry more.

Which one change makes the rollout abort this build before it takes every update?

Question 5 of 7

A chat app's new build changes only how edited messages are saved. Its canary runs at setWeight: 10 with pause: {duration: 30m}, then every request, under a background analysis that measures one metric every 5 minutes over the last 5 minutes. Edits are 2% of the service's requests. Every request fails 0.1% of the time on either build, except edits on the new build: 8% of them fail, each rejected at once with an error. An edit that succeeds takes 0.15 s on either build.

Which metric, put in the template's metrics: slot, makes the analysis abort this build?

Question 6 of 7

A video platform releases a new transcoding-worker build to 50 workers with Argo Rollouts steps setWeight: 10, pause: {duration: 15m}, an inline analysis step, then every worker. The analysis takes one measurement: the number of jobs that failed on the new build's workers since the rollout began, with successCondition 2 or fewer. Each worker starts a new upload as soon as it is free, and every upload takes at least 20 minutes. The uploads the new build's five workers start first take 25, 32, 38, 44 and 50 minutes, and the new build fails every job in its last minute, while packaging.

Which one change makes the rollout abort this build before it reaches every worker?

Question 7 of 7

A ticket shop with 180 venues releases a new checkout build by rings: ring 1, its own test venue, for 2 hours; ring 2, three pilot venues, for 24 hours; ring 3, every venue. Each ring ends with an analysis that aborts the release if checkouts on the new build fail 1% of the time or more. 150 of the venues sell assigned seats; the test venue and the three pilot venues sell general admission only. Each venue sells about 500 tickets a day. The new build fails every checkout that holds an assigned seat.

Which one change makes the release abort while no more than 4 venues have sold through the new build?

Go deeper

Pro tips

Try a new metric in dry-run before it can stop a release. Add it under dryRun in the rollout's analysis; it is measured and reported, but its verdicts do not change the rollout. When: adding a metric whose condition you do not trust yet (step 5). Verify: "A metric running in the dry-run mode won't impact the final state of the rollout or experiment even if it fails or the evaluation comes out as inconclusive." Status: CITED, not run (Argo Rollouts cannot run here).
Give an empty query result a value. Some providers return nothing for a window with no data, and the condition then errors; write successCondition: default(result, 0) < 0.05. When: a metric over a slice or an hour that can see no requests (step 8). Verify: "However, empty query results yielding a nil value can be handled using the default() function." Status: CITED, not run.

Source

Argo Rollouts docs, "Analysis & Progressive Delivery" (the row's first source): https://argo-rollouts.readthedocs.io/en/stable/features/analysis/ · fetched 2026-10-15. "Analysis can be run in the background -- while the canary is progressing through its rollout steps." and "It has no end time, and continues until stopped or failed." (step 4); "startingStep: 2 # delay starting analysis run until setWeight: 40%" (the model's R1); "When analysis is performed "inlined," an AnalysisRun is started when the step is reached, and blocks the rollout until the run is completed." (step 4); "failureCondition can be used to cause an analysis run to fail." (step 5); "Inconclusive analysis runs might also happen when both success and failure conditions are specified, but the measurement value did not meet either condition." (step 5); "A value of 0 for failureLimit means its logic is applicable and no failures are tolerated." (step 6); "The failed analysis causes the Rollout to abort, setting the canary weight back to zero, and the Rollout would be considered in a Degraded." (step 6); "Inconclusive runs causes a rollout to become paused at its current step. Manual intervention is then needed to either resume the rollout, or abort." (step 7); "A metric running in the dry-run mode won't impact the final state of the rollout or experiment even if it fails or the evaluation comes out as inconclusive." (tip 1); "However, empty query results yielding a nil value can be handled using the default() function." (tip 2). The page also calls failureLimit "the maximum number of failed run an analysis is allowed" yet says a run with failureLimit: 3 "is considered as Failed after three failed measurements"; no item rests on a limit above 0. Valid as of 2026-10-15.
Google SRE workbook, ch. 16 "Canarying Releases" (the row's second source): https://sre.google/workbook/canarying-releases/ · fetched 2026-10-15. "Deployments should be performed by computers, not humans." (step 3); "Terminating a canary deployment after receiving just a handful of queries doesn’t provide a useful signal for systems characterized by diverse queries with varied functionality." (step 8); "Performance defects typically manifest only under heavy load" (step 9); "a bad canary could negatively impact the control" (step 10); "It is important to also use absolute measures, such as defined SLOs, to ensure the system is operating correctly." (step 11); "We typically recommend using SLIs as a place to start thinking about canary metrics." and "metrics for CPU usage aren’t as useful: an increase in resource usage doesn’t necessarily impact a service" (step 12). The workbook is a published text Google may revise. Valid as of 2026-10-15.

Checked against the tools and docs as of 2026-10-15.

Predict first