Flaky tests: causes and cures

Make the failure repeat on demand, fix its cause, and never let a retry decide.

Before you read: predict

A ticketing site's seat-hold module and its three tests pass in the file's order:

test('seat 12 starts free', () => assert.equal(isFree(12), true));
test('a held seat is not free', () => { hold(12, 'ana'); assert.equal(isFree(12), false); });
test('a released seat is free again', () => { release(12); assert.equal(isFree(12), true); });

CI now runs every test file in a random order.

In how many of the six orders does a test fail?

const holds = new Map();export const hold = (seat, buyer) => { holds.set(seat, buyer); };export const release = (seat) => { holds.delete(seat); };export const isFree = (seat) => !holds.has(seat);

The 30-second version

A flaky test passes and fails on the same code. The usual causes: state left by another test, waits that guess, the real clock or random values, and live services. Force what varies until the failure repeats on demand, then fix that cause. Meanwhile quarantine the test; a blind retry hides the failure, and some failures are real.

Make the failure repeat on demand, fix its cause, and never let a retry decide.

How it works

Step 1 of 12

Same code, different result. A flaky test passes sometimes and fails sometimes with no change to code, tests or environment; a rerun often passes.

Analogy: A touring magician's card trick

Same props, same deck, same patter, yet the trick fails one night in ten.

On tour	With a flaky test
a card left in the deck by the trick before	state left by an earlier test (step 3)
rehearsing it alone, then straight after each other trick in the set, until it fails on cue	force the orders (step 4)
resetting every prop from the case before each show, not trusting the last trick to put things back	rebuild the starting state (step 5)
rehearsing under hot evening lights on purpose, because the gimmicked card warps in them	the clock, forced (step 7)
the assistant's cue from the wings, lost when the radio drops	a live service (step 9)
the trick cut from the running order for a week while the prop maker rebuilds it	quarantine, with a limit (step 11)
"let's try that again" until it works, hiding a worn gimmick	a blind retry (step 12)

Breaks down when: an audience sees every failure; a red CI run can go unread.

Contrasting case: a magician who counts any trick that worked at the second attempt as a success. The worn gimmick never shows in her notes, so it is never replaced, and it fails at the show that matters (step 12).

In real terms: force what varies until the failure repeats on cue, then remove its cause; never let a retry decide.

Examples, run for real

4.1 ExamHall EXH-5090: the timer test that reads the clock

ExamHall's candidate timer shows the time left as mm:ss. Its test reads the clock to build an end time one hour away, less one second, and the code reads the clock again:

export function timerLabel(endsAtSec) {
  const left = endsAtSec - Math.floor(Date.now() / 1000);
  return String(Math.floor(left / 60)).padStart(2, '0') + ':' + String(left % 60).padStart(2, '0');
}
test('timer shows 59:59', () => {
  const endsAt = Math.floor(Date.now() / 1000) + 3599;
  assert.equal(timerLabel(endsAt), '59:59');
});

4.1 EXH-5090 old test · clock forced to each of the 1,000 start ms of 09:14:59, the code's read 3 ms after the test's · fails at 3 of 1,000 (start ms 997, 998, 999: the reads straddle 09:15:00, label 59:58)

On a real clock the two reads only rarely fall on either side of a second's tick, so the test passes almost every run and a rerun "fixes" it (steps 1, 7).

4.2 MR !5390: inject the clock

The fix passes the time in; the app passes Date.now(), the test a fixed instant. A second test pins the tick the old test tripped over.

4.2 MR !5390 fixed code takes the time; the test passes a fixed one · the same 1,000 forced starts · fails at 0 of 1,000 · and a second test pins the tick: 59:59 at 09:14:59.999, 59:58 one ms later

Forcing the clock turned a one-in-hundreds failure into one that repeats on cue, and the fixed test no longer reads the clock at all, so no start time can fail it (steps 4, 7).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 6

A video app marks an episode "New" on the UTC date it was released. Its code and its one test:

test('an episode out two hours ago is new', () => {
  assert.equal(isNew(Date.now() - 2 * 60 * 60 * 1000), true);
});

The team forces the suite's clock to five start times on one day: 00:30, 01:59, 02:00, 13:00 and 23:30 (UTC).

At how many of the five start times does the test fail?

export function isNew(releasedAtMs) {  const today = new Date().toISOString().slice(0, 10);  return new Date(releasedAtMs).toISOString().slice(0, 10) === today;}

Question 2 of 6

A payments ledger module and its three tests pass in the file's order, but CI runs every file in a random order and most builds go red:

test('a new ledger balances to 0', () => assert.equal(balance(), 0));
test('a first payment gets receipt 1', () => { assert.equal(post({ cents: 500 }), 1); assert.equal(balance(), 500); });
test('a refund nets off part of it', () => { const start = balance(); post({ cents: 500 }); post({ cents: -200 }); assert.equal(balance() - start, 300); });

Which change makes the suite pass in all six orders?

const ledger = [];let nextReceipt = 1;export const post = (entry) => { ledger.push(entry); return nextReceipt++; };export const balance = () => ledger.reduce((sum, e) => sum + e.cents, 0);export const reset = () => { ledger.length = 0; nextReceipt = 1; };

Question 3 of 6

A ticketing site's end-to-end test sends two hold requests for seat 12 at once and expects exactly one to succeed. It fails on some CI runs, and with retries: 2 Playwright reports it flaky. Two explanations are on the table: the test's own timing (it awaits only one of the two replies and assumes which one comes back refused), or a race in the hold endpoint (it checks the seat, then writes the hold, in two steps). Each explanation runs here as a model.

Which observation, from a failing attempt, tells the two explanations apart?

Question 4 of 6

A chat app's team keeps its flaky tests in a quarantine suite outside the main pipeline, under Fowler's numeric limit: at most 8 tests in quarantine. The quarantine holds 8 when a ninth test starts passing and failing on the same code.

What does Fowler say the limit calls for now?

Question 5 of 6

A video app's upload tests save each file under a short random name and never delete it:

test('saves a thumbnail', () => assert.equal(typeof saveTemp('thumb'), 'string'));
test('saves a caption file', () => assert.equal(typeof saveTemp('vtt'), 'string'));
test('saves a preview clip', () => assert.equal(typeof saveTemp('clip'), 'string'));
test('saves a poster', () => assert.equal(typeof saveTemp('poster'), 'string'));
test('saves a transcript', () => assert.equal(typeof saveTemp('txt'), 'string'));

The suite passes on almost every CI run; now and then one test throws "name taken". To chase it, the team stubs Math.random to return 0.5 on every call and runs the tests in file order.

How many of the five tests fail?

const files = new Set();export function saveTemp(bytes) {  const name = Math.random().toString(36).slice(2, 4);  if (files.has(name)) throw new Error(`name taken: ${name}`);  files.add(name);  return name;}

Question 6 of 6

A payments ledger's test 'converts a refund to EUR' fails on some CI builds. CI runs the file in a random order, its log keeps only whether each run passed, not the order it used, and the test calls the live exchange-rate service. Two explanations: another test in the file leaves a stale cached rate behind, or the rate service sometimes fails. Each explanation runs here as a model.

Which next run tells the two explanations apart?

Go deeper

Pro tips

Make a leak fail loudly: give the test database pool one connection and make it throw when it is empty. The first test after the leak then fails, instead of whichever test happens to cross a larger limit. When: the row's typical question "What are the most common causes of flaky tests you have seen?" (step 10). Verify:
```
tip leak in test 2 · pool of 10 · 0 failures in one run · pool of 1 that throws when empty · first failure: test 3 (then every test after it)
```
Status: RAN.

Source

quote-check.mjs finds every passage below verbatim in the saved text. Valid as of 2026-10-18.

Martin Fowler, Eradicating Non-Determinism in Tests, martinfowler.com, 14 April 2011: https://martinfowler.com/articles/nonDeterminism.html (fowler-nondeterminism.txt). The row's first source. Quoted: "A test is non-deterministic when it passes sometimes and fails sometimes, without any noticeable change in the code, tests, or environment." · "Such tests fail, then you re-run them and they pass." · "The trouble with non-deterministic tests is that when they go red, you have no idea whether its due to a bug, or just part of the non-deterministic behavior." · "Once that discipline is lost, then a failure in the healthy deterministic tests will get ignored too." · "If you have non-deterministic tests keep them in a different test suite to your healthy tests." · "So for most teams I'd advise keeping the quarantined tests out of the main pipeline." · "If one test creates some data in the database and leaves it lying around, it can corrupt the run of another test which may rely on a different database state." · "Properly isolated tests can be run in any sequence." · "With clean-up, however, one test will contain the bug, but another test will fail - so it's hard to find the real problem." · "In particular be aware with static data and singletons." · "Never use bare sleeps to wait for asynchonous responses: use a callback or polling." · "Few things are more non-deterministic than a call to the system clock." · "Always wrap the system clock, so it can be easily substituted for testing." · "One of my colleagues likes to force a test run just before and after midnight in order to catch tests that use the current time and assume it's the same day an hour or two later." · "one way to manifest it is to stub the randomizer for testing so it always returns the same value." · "Often remote systems don't have test system we can call, which means hitting a live system." · "it's time to reach for a Test Double - a component that looks like the remote service, but is really just a pretend version that mimics the remote system's behavior." · "If your application has some kind of resource leak, this will lead to random tests failing, since it's just which test causes the resource leak to go over a limit that gets the failure." · "a good tactic is to configure the pool to a size of 1 and make it throw an exception should it get a request for a resource when it has none left to give." Supports steps 1–12.
Playwright, Retries, playwright.dev: https://playwright.dev/docs/test-retries (pw-test-retries.txt). The row's second source. Quoted: "Test retries are a way to automatically re-run a test when it fails." · "Playwright Test will discard the entire worker process along with the browser and will start a new one." · ""flaky" - tests that failed on the first run, but passed when retried;" · "It is usually better to make your tests isolated, so they can be efficiently run and retried independently." Supports steps 1 and 12.

Read first: Playwright end-to-end tests that hold up

Checked against the tools and docs as of 2026-10-18.

Predict first