Skip to content

Desired state: infrastructure as code and Kubernetes

Declare what should exist; the tool diffs it against what exists and acts. Readiness routes traffic; liveness restarts.

Predict first

Before you read: predict

A ticketing platform manages its API firewall with Terraform. The configuration allows port 443 only, and the last terraform apply, a week ago, left the firewall that way. On Friday, during an incident, an engineer added port 22 to the firewall by hand in the cloud console. On Monday a teammate adds a DNS record to the configuration, runs terraform apply and approves the plan it shows. The provider can change a firewall's ports in place.

On this lesson's Terraform model, what does Monday's apply do?

Pick one

The 30-second version

Declarative tools take a description of what should exist and make the real world match it. Terraform's plan reads the objects it manages, compares them with the configuration and lists the changes; apply makes them. Kubernetes controllers repeat that comparison forever. Readiness decides whether a pod gets traffic; liveness decides whether its container restarts.

Declare what should exist; the tool diffs it against what exists and acts. Readiness routes traffic; liveness restarts.

How it works

Step 1 of 16

Desired state. You write what should exist: a Terraform configuration, a Kubernetes manifest. The tool works out the steps.
In the kitchen
  • the patrol plan on the hut wall → the configuration or manifest (step 1)

Analogy: A beach lifeguard service, its patrol plan and the supervisor's sweeps

Each morning the patrol plan on the hut wall says which towers are staffed; the supervisor drives the beach all day, checking it against the plan.

On the beach In the system
the patrol plan on the hut wall the configuration or manifest (step 1)
the supervisor's sweep, comparing each tower with the plan a plan, or a controller's loop (steps 3, 9)
a guard sent to a tower the plan staffs and nobody holds a create (steps 5, 10)
a red "not on watch" flag while a guard changes over: no swimmers sent to that zone, the guard stays readiness failing (step 13)
three missed radio checks in a row: the guard is stood down and a fresh one takes the tower liveness, and a restart (steps 12, 14)
a new guard's warm-up swim before her first radio check a startup probe (step 15)

In real terms: a declarative tool keeps comparing what you wrote with what runs and closes the gap; probes tell the cluster which pods may serve and which to restart.

Breaks down when: the supervisor can ask why a check was missed and wait out a radio fault; the kubelet acts on the count alone.

Contrasting case: "send two more guards", phoned to the agency twice by mistake: four guards arrive. That is a script (step 2).

Examples, run for real

4.1 ExamHall EXH-4770: a new pod took exam traffic while it warmed

EXH-4770: ExamHall's exam API pods had a liveness probe on /livez, which answers while the process runs, and no readiness probe. A deploy replaced one of three pods; the new pod spends 20 s warming its question cache, and requests it took meanwhile answered "question unavailable". MR !5070 adds a readiness probe on /ready, which answers 200 once the cache is warm; liveness still checks only the process.

4.1 EXH-4770 no readiness probe   exam requests 180 · "question unavailable" 19 · new pod in the Service from 0 s · restarts 0
4.1 MR !5070 readiness on /ready  exam requests 180 · "question unavailable" 0 · new pod in the Service from 20 s · restarts 0

With no readiness probe the new pod counted as ready at once and took its share cold; with one it joined the Service at its first passing probe, and no container restarted (steps 11, 13, 16).

4.2 Ticketing: a Deployment keeps its count

The seat-holds Deployment is raised from 3 to 5 replicas; later a node fails, taking 2 of its pods with it.

4.2 replicas 3 -> 5             running before 3 · pods created 2 · running 5
4.2 node lost with 2 pods       running 3 · pods created 2 · running 5

Each time, the ReplicaSet compared its count with the pods it found and created the difference (steps 9–10).

4.3 Ticketing: a change that cannot be made in place

The holds queue becomes a FIFO queue, a setting its provider cannot change on an existing queue.

4.3 holds: fifo false -> true   -/+ queue.holds (replace) · Plan: 1 to add, 0 to change, 1 to destroy. · old queue gone yes · new id i-100

The plan destroys the old queue and creates a new one with a new id (step 7).

Picture it

a charcoal conductor's baton resting on a lavender stand; nothing is labelled.
An illustration to help it stick. The mechanism is in the sections above.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

The ticketing platform's Terraform state records three queues it created: orders, refunds and emails. A fourth queue, scratch, was made in the cloud console and never imported. A merge request changes the configuration: it deletes the emails block, renames the refunds block to refund_requests (the queue's name with it), raises orders' message retention from 4 to 14 days, and adds a waitlist queue. The provider can change a queue's retention in place.

On this lesson's Terraform model, what summary line does terraform plan print?

Pick one

Question 2 of 6

The seat-map service is a Deployment with replicas: 3. Its ReplicaSet's selector is app: seatmap, and its three pods, seatmap-a, seatmap-b and seatmap-c, carry that label. While debugging, an engineer does three things, each time waiting until the ReplicaSet has acted: she deletes seatmap-a; she starts a pod by hand from the same image, labelled app: seatmap, to attach a debugger; she changes seatmap-b's label to app: seatmap-debug.

At the end, on this lesson's ReplicaSet model, how many pods labelled app: seatmap are running, and how many pods are running in all?

Pick one

Question 3 of 6

The box office's API runs 3 pods, all started at 20:00:00. Each has a liveness probe: httpGet on /healthz, initialDelaySeconds: 10, periodSeconds: 10, failureThreshold: 3. /healthz runs SELECT 1 against the database and answers 500 when it fails. So every pod is probed at 20:00:10, 20:00:20 and so on, and a restarted container is first probed 10 s after its restart. Tonight the database is unreachable three times: from 20:01:05 to 20:01:28, from 20:11:05 to 20:11:38, and from 20:21:05 to 20:21:52.

On this lesson's kubelet model, how many container restarts happen across the 3 pods by 20:23:00?

Pick one

Question 4 of 6

The ticket-search API loads its search index before its HTTP server listens: 65 s on a warm node, 145 s on a cold one. Its probes: a readinessProbe on /ready (periodSeconds: 10, failureThreshold: 3) and a livenessProbe on /livez with initialDelaySeconds: 70, periodSeconds: 10, failureThreshold: 3; /livez answers 500 once the request loop stops making progress. On a cold node the pod is restarted again and again and never serves. Any probe added runs every 10 s (periodSeconds defaults to 10). The team's checks: the pod must start on either node, and on a warm node whose request loop stops 100 s after start, the kubelet must restart it within 30 s.

On this lesson's kubelet model, which one change passes both checks?

Pick one

Question 5 of 6

The seat-map API runs 3 pods behind a Service. Each serves the seat map from its in-memory cache and sells tickets through the database; a new pod fills its cache from the database in 20 s. Its probes: a livenessProbe on /livez, which answers 200 while the process runs, and a readinessProbe on /ready, which answers 200 only when the cache is full and SELECT 1 succeeds; both periodSeconds: 5, failureThreshold: 3. Database failovers usually take 40 s and can take 3 minutes, and next week a node drain will replace one pod 1 minute into the failover.

On this lesson's kubelet and Service model, which one change keeps every seat-map request served through a 3-minute failover, with no container restarted?

Pick one

Question 6 of 6

During Saturday's on-sale, the on-call engineer made three changes by hand in the cloud console: she raised the ticket database's max_connections from 200 to 400, on purpose, raised the holds queue's retention from 4 to 7 days, and created a queue named overflow. Terraform manages the database and holds; it never recorded overflow. On Monday the team edits the configuration to set max_connections to 400, changes nothing else, then runs terraform apply -refresh-only and approves it. On Tuesday a teammate adds a DNS record to the configuration and runs terraform plan. The provider can change both settings in place.

On this lesson's Terraform model, what summary line does Tuesday's plan print?

Pick one

Go deeper

Pro tip
  1. Run terraform plan -detailed-exitcode on a schedule: exit code 2 means the real objects no longer match the configuration, so a console change shows up before someone's next apply undoes it. When: a team whose on-call sometimes changes things by hand (steps 3, 8). Verify:
    tip1 nightly plan, nothing changed by hand      exit code 0
tip1 nightly plan, port 22 added in the console exit code 2
Sources

Vendor docs change, so each is valid as of 2026-10-17.

  • HashiCorp Terraform docs, The core Terraform workflow (named in the row): https://developer.hashicorp.com/terraform/intro/core-workflow · tf-core-workflow.txt. "if a manual change was made to your infrastructure since the plan was reviewed, the plan might be different when you merge." Supports steps 1, 3–4.
  • HashiCorp Terraform docs, terraform plan command: https://developer.hashicorp.com/terraform/cli/commands/plan · tf-plan-command.txt. "Reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date."; "Proposes a set of change actions that should, if applied, make the remote objects match the configuration."; "setting refresh=false causes Terraform to ignore external changes"; "2 = Succeeded with non-empty diff (changes present)". Supports steps 3, 8 and the tip.
  • HashiCorp Terraform docs, Resources (the resource behaviour page redirects here): https://developer.hashicorp.com/terraform/language/resources · tf-resource-behavior.txt. "Creates resources in the configuration that do not yet exist as real infrastructure objects."; "Destroys resources that exist in the state but no longer exist in the configuration."; "Destroys and re-creates resources whose arguments have changed but cannot be updated in-place due to remote API limitations."
  • Kubernetes docs, Controllers (named in the row): https://kubernetes.io/docs/concepts/architecture/controller/ · k8s-controller.txt. "Each controller tries to move the current cluster state closer to the desired state." Supports step 9.
  • Kubernetes docs, ReplicaSet: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ · k8s-replicaset.txt. "A ReplicaSet then fulfills its purpose by creating and deleting Pods as needed to reach the desired number."; "If there is a Pod that has no OwnerReference or the OwnerReference is not a Controller and it matches a ReplicaSet's selector, it will be immediately acquired by said ReplicaSet."; "Pods that are removed in this way will be replaced automatically".
  • Kubernetes docs, Liveness, Readiness, and Startup Probes (named in the row; the concept page redirects here): https://kubernetes.io/docs/concepts/workloads/pods/probes/ · k8s-probes-concept.txt. "After a probe fails failureThreshold times in a row, Kubernetes considers that the overall check has failed: the container is not ready/healthy/live."; "controller removes the Pod's IP address from the EndpointSlices of all Services that match the Pod."; "Liveness probes do not wait for readiness probes to succeed."; "If a startup probe is configured, Kubernetes does not execute liveness or readiness probes until the startup probe succeeds"; "If a startup probe is defined, liveness and readiness probe delays do not begin until the startup probe has succeeded."; "the existence of the readiness probe in the spec means that the Pod will start without receiving any traffic and only start receiving traffic after the opening question starts succeeding."; "The liveness probe passes when the app itself is healthy, but the readiness probe additionally checks that each required back-end service is available. This helps you avoid directing traffic to Pods that can only respond with error messages."; "Incorrect implementation of liveness probes can lead to cascading failures."
  • Kubernetes docs, Configure Liveness, Readiness and Startup Probes (named in the row): https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ · k8s-probes-task.txt. "If the readiness probe fails, the pod will be marked unready and will not receive traffic from any services." Supports steps 11, 13.
  • Kubernetes docs, Service: https://kubernetes.io/docs/concepts/services-networking/service/ · k8s-service.txt. "to forward traffic to one of the ready endpoints associated with that Service". Supports step 11.
  • Also saved, not quoted: the Deployment and Pod Lifecycle pages (k8s-deployment.txt, k8s-pod-lifecycle.txt), read for the restart policy a Deployment's pods use.

Related lessons

Checked against the tools and docs as of 2026-10-17.