Incident response and on-call

An Earth-observation satellite's battery starts to overheat.

At mission control	In an incident
the flight director holds the picture and gives each controller a job, but sends no commands herself	the incident commander (steps 3–4)
one controller at the command console sends every command	one set of hands (step 5)
a liaison tells the imaging customers, on the hour, that pictures are paused	updates on a cadence (step 6)
safe mode ordered at once, panels to the sun and instruments off, knowing only that the fault is in the power system	mitigate first, on location alone (steps 7–8)
the satellite kept in safe mode until the heater fault is found	mitigated is not fixed (step 9)
the shift handover read aloud and acknowledged	handing over aloud (step 10)
the anomaly review asking what let a wrong heater setting be uploaded, not who typed it	a blameless postmortem (step 11)
console shifts with rest days between them	sustainable on-call (steps 12–13)

In real terms: one person directs, one team acts, users hear on a schedule, impact stops before the cause is known, and the review fixes the system.

Breaks down when: safe mode is built into a satellite and rehearsed for years. A service's mitigations exist only if someone listed and rehearsed them before the incident, as runbook entries (step 8).

Contrasting case: an orbit change planned weeks ahead gets a reviewed procedure, not a flight director improvising. That is a planned change (judgment-change-safety).

4.1 ExamHall EXH-2200: the same incident, unmanaged and managed

EXH-2200: build 8.2 went out at 08:40. From 09:01, under exam load, 1 autosave in 4 answered 502, with 2,400 candidates saving twice a minute. Three engineers investigated from their own terminals; at 09:31 the pool metrics led Ines to a missing answers.attempt_id index, and the rollback to 8.1 finished at 09:36. MR !2500 is the runbook replayed below: Dana declares at 09:04 and commands, changing nothing; Sami, the operations lead, starts the six-minute rollback at 09:05; Ines keeps investigating.

4.1 as handled no commander          · rollback done 09:36 · root cause 09:31 · user impact 09:01-09:36 · failed autosaves 42000 · after the root cause was found 6000 · first word to institutions 09:40
4.1 managed    commander from 09:04  · rollback done 09:11 · root cause 09:31 · user impact 09:01-09:11 · failed autosaves 12000 · after the root cause was found    0 · updates to institutions 09:06 09:21 09:36

The root cause arrives at 09:31 in both runs. Managed, users stopped failing twenty minutes before it, because the mitigation did not wait for the cause (step 7).

4.2 Proof domain, checkout: a week of night shifts

The checkout team's night on-call took these incidents in one week, at six hours of work each, root cause and postmortem included (step 13):

4.2 checkout nights   incidents 1 0 3 0 2 0 1 · hours of incident work 6 0 18 0 12 0 6 · shifts with more work than their 12 hours: 1 (Wed) · median incidents per shift 1

Wednesday needed eighteen hours of work in a twelve-hour shift, though the median night had one incident. A rotation can be small in share yet still overloaded in quality, so both are counted.