Testing judgment: confidence per cost
Spend tests where failure is likely and costly, keep the risky part real, assert what callers see, and see each test fail.
Predict first
Before you read: predict
ExamHall's admin console ships at 17:00. The testing time left covers two of these five areas, about an hour each. For each area: the releases (of the last 20) in which it failed in production, the support team's estimate of what one such failure costs, and its line coverage today.
area failed releases cost of one failure line coverage
results publish 4 of 20 £20,000 91%
CSV export headers 6 of 20 £200 38%
audit-log retention 1 of 20 £40,000 96%
late-penalty marks 5 of 20 £10,000 64%
theme settings 1 of 20 £100 12%
Which two areas get the testing time?
The 30-second version
Testing time is finite, so rank where a failure is both likely and costly: likelihood from where failures happened, impact from what one costs, multiplied. Test that area with its risky part running for real, at the cheapest level that does so. Assert what callers see, not how the code is built. Watch each new test fail once.
Spend tests where failure is likely and costly, keep the risky part real, assert what callers see, and see each test fail.
How it works
Step 1 of 12
Analogy: A home inspector's half-day inspection of an old house before its purchase
The inspector has one morning and cannot open every wall.
| On the inspection | In the test plan |
|---|---|
| the old roof and the 1970s wiring, likely to fail and costly to repair, get the morning | likelihood times impact (steps 2–4) |
| the paint colours and door handles get a glance | cheap, rarely failing code (steps 5, 12) |
| climbing into the loft instead of judging the roof from the street, without hiring scaffolding | the risky part real, at the cheapest level (steps 6–7) |
| running a hot tap and feeling the radiators warm, not reading the boiler's service sticker | asserting what the caller sees (steps 8–9) |
| checking the damp meter on a wet sponge first, so a dry reading means something | seeing a test fail once (steps 10–11) |
In real terms: rank areas by how likely and how costly a failure is, run the risky part for real, assert the outcome, and see each test go red once.
Breaks down when: an inspection happens once, while a suite reruns on every change, and its likelihoods move with each release's diff.
Contrasting case: a checklist that gives every room the same ten minutes spreads effort evenly, whatever each room risks.
Examples, run for real
4.1 ExamHall EXH-2120: where the tests sat
EXH-2120: the admin console's results release shipped at 95% line coverage, and a re-grade then published stale marks to ECON-201 candidates. The release's 61 new tests, against each area's last 20 releases:
4.1 results publish tests 3 · coverage 96% · escapes 4 of 20 releases · £20,000 each
4.1 re-grade tests 2 · coverage 93% · escapes 3 of 20 releases · £15,000 each
4.1 column labels and buttons tests 40 · coverage 100% · escapes 0 of 20 releases · £100 each
4.1 filters and sorting tests 16 · coverage 88% · escapes 1 of 20 releases · £500 each
4.1 suite: 61 tests · line coverage 95% (380/400) · 7 of 8 escapes in results publish and re-grade, which hold 5 of 61 tests
Seven of eight escapes came from two areas holding five tests, while the 40 label tests sat where nothing failed and little was at stake; full coverage there hid the gap (steps 2–5). MR !2420 moves the hours to publish and re-grade and drops the label tests.
4.2 Proof domain, checkout: a green test on a wrong fee
The delivery fee comes from rates.zoneRate(zone). One test checks that the helper was called with 'local'; another checks the fee a £30 basket pays. A slip then makes zoneRate return pounds instead of pence.
4.2 fee for a £30 local basket: before 399 · after the slip 3.99
4.2 call check (zoneRate called with 'local'): before passes · after passes
4.2 fee check (a £30 local basket pays 399): before passes · after fails
The call check stays green while the customer is charged £0.04: it reads which helper ran, not what the caller gets (steps 8–9).
Try it
Answer first, then check. Nothing is saved.
Question 1 of 5
The checkout's basket and four of its tests. Each test starts with const b = new Basket(); b.add('mug', 1200, 2); b.add('tea', 450, 1);.
// basket.test.mjs
it('a', () => { expect(b.total('TEN')).toBe(2565); });
it('b', () => { expect(b.lines).toEqual([{ sku: 'mug', pricePence: 1200, qty: 2 }, { sku: 'tea', pricePence: 450, qty: 1 }]); });
it('c', () => { const spy = spyOn(pricing, 'discount'); b.total('TEN'); expect(spy.calls).toEqual([[2850, 'TEN']]); });
it('d', () => { expect(b.lines.length).toBe(2); expect(b.total('TEN')).toBe(2565); });
All four pass today. A refactor then changes how Basket is built, and changes no total it returns:
// basket.mjs (after the refactor; pricing unchanged)
export const pricing = {
lineTotal: (line) => line.pricePence * line.qty,
discount: (pence, code) => (code === 'TEN' ? Math.round(pence / 10) : 0),
};
export class Basket {
constructor() { this.qty = new Map(); this.price = new Map(); }
add(sku, pricePence, qty) { this.price.set(sku, pricePence); this.qty.set(sku, (this.qty.get(sku) ?? 0) + qty); }
total(code) {
let sub = 0;
for (const [sku, n] of this.qty) sub += this.price.get(sku) * n;
return code === 'TEN' ? sub - Math.round(sub / 10) : sub;
}
}
Which tests fail after the refactor?
// basket.mjs (today)export const pricing = { lineTotal: (line) => line.pricePence * line.qty, discount: (pence, code) => (code === 'TEN' ? Math.round(pence / 10) : 0),};export class Basket { constructor() { this.lines = []; } add(sku, pricePence, qty) { this.lines.push({ sku, pricePence, qty }); } total(code) { const sub = this.lines.reduce((s, l) => s + pricing.lineTotal(l), 0); return sub - pricing.discount(sub, code); }}Question 2 of 5
The checkout's gift-card service calls one repository method, which runs this on Postgres:
UPDATE gift_cards SET balance_pence = balance_pence - $1
WHERE code = $2 AND balance_pence >= $1
RETURNING balance_pence;
Both gift-card bugs that reached customers last quarter were in that SQL: one let a card go below zero, and one charged a card that had been cancelled. The team has a day for gift-card tests.
Which test do they write first?
Question 3 of 5
Last week a three-item return refunded VAT on the first item only. An MR fixes refundVat() and adds refund.test.mjs, and CI is green on the MR.
Before approving, which check shows that the new test guards against that bug coming back?
Question 4 of 5
An MR adds three parts to the checkout: countryName(code), which returns COUNTRIES[code] from a table generated from the ISO 3166 list; giftCardRounding(pence), copied unchanged from the old checkout, where it ran for a year with no reported bug; and the branch that refunds the card when the payment provider times out, which runs about once a week. The MR adds three tests of countryName(), one of giftCardRounding() and none of the timeout refund.
Which comment does the reviewer leave?
Question 5 of 5
The basket API's reply and the web client's total label:
Two MRs are coming. MR 1 changes the reply to { total: (basket.totalPence / 100).toFixed(2) } and does not touch the client. MR 2 changes it to { totalPence: basket.totalPence, currency: 'GBP' }. The team adds one test now, before either MR, and wants it to fail on MR 1 and pass on MR 2.
it('A', () => { expect(totalLabel({ totalPence: 4599 })).toBe('£45.99'); });
it('B', () => { expect(totalLabel(basketBody({ totalPence: 4599 }))).toBe('£45.99'); });
it('C', () => { expect(basketBody({ totalPence: 4599 })).toEqual({ totalPence: 4599 }); });
Which test?
// api/basket-body.mjsexport function basketBody(basket) { return { totalPence: basket.totalPence }; }// web/total-label.mjsexport function totalLabel(body) { return `£${(body.totalPence / 100).toFixed(2)}`; }Go deeper
Pro tips
- For every comparison in money or grades code, test the boundary value itself and one step on each side. That one habit catches a misplaced or mistyped comparison that the boundary and the value just past it can both miss. When: writing tests for a pass mark, a free-delivery threshold or a refund limit. Verify:
tip x <= 10 typed as x === 10 · inputs 10, 11: caught by none · inputs 9, 10, 11: caught by 9
Source
The ISTQB syllabus is a PDF, turned into text with pdfjs-dist (extract-pdf.mjs; its file carries a Via line). Valid as of 2026-10-01.
- "In the quantitative approach the risk level is calculated as the multiplication of risk likelihood and risk impact."; "Defects cluster together."; "BVA focuses on the boundary values of the partitions because developers are more likely to make errors with these boundary values." Supports steps 1–4, the opening question, t2 and the pro tip.
- Martin Fowler, Test Coverage (2012): https://martinfowler.com/bliki/TestCoverage.html (
fowler-test-coverage.txt). "lots of tests looking for things that rarely go wrong distracting you from testing the things that really matter". Supports step 5 and the opening question. - Martin Fowler, Self Testing Code (2014): https://martinfowler.com/bliki/SelfTestingCode.html (
fowler-self-testing.txt). "first write a test that exposes the bug, and only then to try to fix it"; "any non-trivial code without tests is broken". - Kent C. Dodds, Write tests. Not too many. Mostly integration.: https://kentcdodds.com/blog/write-tests (
dodds-write-tests.txt). "When you mock something you're removing all confidence in the integration between what you're testing and what's being mocked."; "Things that really have no logic in them at all". - Kent C. Dodds, Testing Implementation Details: https://kentcdodds.com/blog/testing-implementation-details (
dodds-impl-details.txt). "Implementation details are things which users of your code will not typically use, see, or even know about."; such tests "Can break when you refactor application code" and "May not fail when you break application code". Supports steps 8–9. - Titus Winters, Tom Manshreck and Hyrum Wright (eds.), Software Engineering at Google, ch. 11 "Testing Overview" and ch. 13 "Test Doubles": https://abseil.io/resources/swe-book/html/ch11.html and https://abseil.io/resources/swe-book/html/ch13.html (
swe-ch11.txt,swe-ch13.txt). Fidelity from real implementations; larger tests as sanity checks; coverage measures invocation, "not what happened as a result". Google's stated practice, as of 2026-10-01. - Kent Beck, Canon TDD (2023): https://tidyfirst.substack.com/p/canon-tdd (
beck-canon-tdd.txt), and Test Desiderata: https://kentbeck.github.io/TestDesiderata/ (beck-desiderata.txt). The copied-expected-value mistake; listing variants such as a timeout; "If the behavior changes, the test result should change."
Related lessons
Checked against the tools and docs as of 2026-10-01.