Blameless postmortems
Timeline on one clock, impact from the users' side, several causes, and owned actions that change the system.
Predict first
Before you read: predict
On Tuesday afternoon ExamHall's proctors watched a frozen live feed during ECON-110, and the postmortem draft needs the impact duration. The feed service's request log shows failed feed updates from 14:21 until 14:43, when updates succeeded again. The feed-errors alert fired at 14:26 and resolved at 14:51. The incident was declared at 14:28 and closed at 15:10.
How many minutes of impact should the postmortem state?
The 30-second version
A postmortem is the written record of an incident: a timeline from every source on one clock, the impact as users felt it, the conditions that let it happen, and actions that change the system, each owned and followed to done.
Timeline on one clock, impact from the users' side, several causes, and owned actions that change the system.
How it works
Step 1 of 13
Analogy: A school trip review (a pupil left behind at a museum)
The coach left a museum with one pupil still inside. The next day the head of year writes it up.
| At the school | In a postmortem |
|---|---|
| the coach's GPS log (in UTC), the museum's door log and each teacher's own account, put on one clock | a timeline from every source (steps 2–3) |
| the pupil waited 40 minutes, from the coach leaving to a teacher's return, not from the first phone call | impact from the users' side (steps 4–5) |
| the register still listed a pupil sent home sick, the meeting point was by a busy exit, the coach had a fixed slot | several causes (step 7) |
| the teacher who counted helps write the changes | the people involved help (step 10) |
| "count by name at the coach door: Mr Hale, before the next trip" | an owned action that changes the system (steps 9, 11) |
In real terms: one merged timeline, an impact measured on the users, a cause list, and dated tickets with owners.
Breaks down when: the school can ask everyone the same evening. A system's logs roll over and people's memories fade, so a postmortem gathers both while they last (step 2).
Examples, run for real
4.1 ExamHall EXH-4010 → MR !4310: the exam start delayed
EXH-4010 is the postmortem for ECON-201's delayed start, on synthetic records from five sources. MR !4310 is its first action.
4.1 EXH-4010 · ECON-201 · timeline on one clock (CEST, UTC+2), merged from 5 records
07:58 warm job writes the 4 ECON-201 paper keys, TTL 60 min [cron log, CEST 07:58]
08:55 lobby opens: 2,400 candidates' browsers load the paper [exam schedule, CEST 08:55]
08:58 the 4 paper keys expire together; first paper load fails (pool wait) [Redis and app logs, UTC 06:58]
09:04 start-error alert fires (it needs 5 minutes of errors) [alerting, UTC 07:04]
09:05 incident declared [incident channel, CEST 09:05]
09:09 pre-warm run by hand [incident channel, CEST 09:09]
09:11 last failed paper load [app log, UTC 07:11]
09:12 papers load again; the exam start moves to 09:14 [app log, UTC 07:12]
09:20 start-error alert resolves [alerting, UTC 07:20]
09:31 incident closed [incident channel, CEST 09:31]
sorted as printed instead, the expiry (06:58) lands before the warm job (07:58 CEST) that wrote its keys
impact 08:58-09:12 · 14 minutes · 2,400 candidates' starts delayed (the alert was up 16 minutes, the incident open 26: neither is the impact)
causes 1 one job wrote all 4 keys with one 60-minute TTL, so they expired together 3 minutes after the lobby opened
2 each miss rebuilt the paper from Postgres on its own (no single-flight), so 2,400 loads queued for 20 connections
3 nothing watched pool waits, and the only alert needed 5 minutes of start errors, so the first page came at 09:04
actions MR !4310 single-flight on the paper key · prevent · Mia · OPS-212 · due 2026-10-20 (cause 2)
pre-warm 10 minutes before each lobby opens, TTLs staggered · prevent · Dev · OPS-213 · due 2026-10-27 (cause 1)
alert on pool wait over 1 s for 1 minute · detect · Ana · OPS-214 · due 2026-10-20 (cause 3)Read as text, the five records put the keys' expiry before the job that wrote them; on one clock the story runs in order (step 3). The impact is 14 minutes from the candidates' own requests, not the alert's 16 or the incident's 26 (steps 4–5). No cause is a person: each is a condition, and each gets its own owned action (steps 7, 9, 11).
4.2 Ticketing: a draft action table, checked
A ticketing site charged some fans twice. Its postmortem draft lists four actions; actions-lint.mjs checks each for a system change, an owner, a ticket, a due date and a type:
4.2 ticketing · duplicate-charge postmortem · 4 draft actions · actions-lint
FAIL "engineers to be more careful with payment retries" · changes no system
FAIL "investigate the payment gateway's timeouts" · no owner · no ticket · no due date · no type
OK "idempotency key on every charge call" · prevent · Lea · PAY-88 · due 2026-10-24
OK "unique key on order id in the charges table" · prevent · Sam · PAY-89 · due 2026-10-31The first line has an owner and a date but changes nothing (step 12); the second is a task with nobody on it (step 11). The two that pass each change a system.
Try it
Answer first, then check. Nothing is saved.
Question 1 of 9
A concert on-sale's postmortem merges five records, one line from each source. Each line is printed as its record keeps it:
- alerting (UTC): 09:06 the hold-expiry alert fires
- support desk (CET, UTC+1): 10:04 the first fan email arrives
- deploy log (UTC): 08:52 deploy 3.7 lowers the seat-hold TTL
- hold service (UTC): 09:01 the first seat holds expire at payment
- queue service (CET, UTC+1): 09:58 the waiting room opens
Put the five events in the order they happened, the earliest first.
Your order
- 1empty
- 2empty
- 3empty
- 4empty
- 5empty
Tap the lines in the order you expect.
Question 2 of 9
A ticketing site's seat map went blank for 9 minutes when its pricing service ran out of database connections. The postmortem lists two actions: fix the pricing service's connection leak, and have the seat map show the last price snapshot, marked as such, when pricing does not answer within 2 seconds. A reviewer wants the second struck, since it would not stop the pricing service failing.
What should the author do?
Question 3 of 9
At 10:02 an engineer on a ticketing site ran the manual price import with last season's file, and 120 seats went on sale at €0; she saw it at 10:08 and re-ran the import with the right file. The review found that the importer accepts a €0 price, that the import writes straight to the live seat map with no preview, and that no alert watches for €0 sales.
Which cause list should the postmortem carry?
Question 4 of 9
In March, ExamHall's autosave postmortem ended with four actions, each filed as a ticket with a named owner and a due date in April, and the review signed it off. In June the same autosave stall happened again. All four tickets were still open in the backlog: at every sprint planning, the feature tickets had been picked first.
What should the team change?
Question 5 of 9
For P1's on-sale, the review agreed three causes: deploy 3.7 cut the seat-hold TTL to 2 minutes with no check against payment time; holds expired while fans were inside the bank's 3-D Secure step; the hold-expiry alert fired after the first fan email.
Write one action per cause as a JSON row with what, factor, type, owner, ticket and due.
Question 6 of 9
An interviewer tells you that an engineer's command caused an outage and asks how you would write it up.
Answer in at most 150 words, using an incident you ran or saw.
Question 7 of 9
An interviewer asks you to walk them through a postmortem you wrote or ran.
Write your answer in at most 150 words, as you would say it.
Question 8 of 9
During a concert on-sale, about 50 checkouts arrived every minute. The checkout service's log gives the failed and the arrived checkouts per minute (CET): 10:00 48/48 · 10:01 50/50 · 10:02 47/47 · 10:03 12/49 · 10:04 0/51 · 10:05 0/50 · 10:06 0/48 · 10:07 0/50 · 10:08 50/50 · 10:09 46/46 · 10:10 5/49 · 10:11 0/50. In every later minute no checkout failed. The status page said "investigating" at 10:02 and "resolved" at 10:20.
How many minutes of impact go in the postmortem?
Question 9 of 9
The timeline for ECON-110's frozen proctor feed, built from logs, reads: "14:30 Omar restarts the feed service. 14:34 Omar restarts the feed service again." The review asks what the 14:34 entry still needs.
What should be added?
Go deeper
Pro tip
- Open the timeline document during the incident, and have the communications lead paste each update into it with its time. Most of the timeline then exists before anyone sits down to write. When: at declaration. Status: SELF (practice target; not run).
Source
- S1 · Google SRE book, "Postmortem Culture: Learning from Failure" (John Lunney and Sue Lueder), the row's first source, https://sre.google/sre-book/postmortem-culture/ (
sre-postmortem-culture.txt). "A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring." (step 1); "that all contributing root cause(s) are well understood, and, especially, that effective preventive actions are put in place to reduce the likelihood and/or impact of recurrence." (steps 7, 9); "it should call out where and how services can be improved." (step 9); "the postmortem format clearly identifies the actions that led to the incident" (steps 7–8); "Are the impact assessments complete?" and "Is the action plan appropriate and are resulting bug fixes at appropriate priority?" (step 13); "An unreviewed postmortem might as well never have existed." (step 13). - S2 · John Allspaw, "Blameless PostMortems and a Just Culture" (Etsy Code as Craft, 22 May 2012), the row's second source, https://codeascraft.com/2012/05/22/blameless-postmortems/ (
allspaw-blameless-2012.txt). etsy.com answered 403 (a bot check) on 2026-10-13, so the text is the Wayback Machine's capture of the original URL taken on 2013-06-09, read through https://web.archive.org/web/20130609224956id_/http://codeascraft.com/2012/05/22/blameless-postmortems/ (the file'sVia:line); the comments under the post are part of the page, and the reply marked as the post author's (jallspaw, 13 June 2012) is quoted as Allspaw's reply under the post, on Etsy's practice. "can give a detailed account of: what actions they took at what time, what effects they observed, expectations they had, assumptions they had made, and their understanding of timeline of events as they occurred." (steps 2, 8); "Saying what people should have done doesn’t explain why it made sense for them to do what they did" (step 8); "We seek out Second Stories, gather details from multiple perspectives on failures, and we don’t punish people for making mistakes." (step 2); "They ought to be heavily involved in coming up with remediation items." (step 10); "We enable and encourage people who do make mistakes to be the experts on educating the rest of the organization how not to make them in the future." (step 10); "Human error is seen as the effect of systemic vulnerabilities deeper inside the organization" (a second story; step 7); "Telling people to be more careful will make the problem go away" (listed among the first stories, which the post rejects; step 12); from Allspaw's reply under the post, on Etsy's practice: "Not acting on remediation items is akin to paying lip-service and essentially dismissing the learning out of hand." (step 11); "we have a 30 days (this is likely too long) due date on remediation tasks to get done, and they are essentially actionable tickets that trump any other work that the engineer is currently working on" (step 11).
Related lessons
Checked against the tools and docs as of 2026-10-13.