Incident: flaky end-to-end suite

When a suite turns flaky as workers go up, read what the assertion received, give each worker its own data, then fix the wait.

Before you read: predict

CI · main red three times today · candidate-autosave.spec.ts › shows Saved after typing failed 9 of the last 110 runs · the release train is blocked

Last week ExamHall's CI went from 1 Playwright worker to 4, which cut it from 18 to 6 minutes. The seed file creates one candidate with one attempt, a-1, and every spec uses it. The CI machines run at full CPU, so in this lesson's busy-machine model a save's reply can take longer than a second. While a save is in flight the status reads "Saving your answer…"; when the server refuses a save, the page shows "Already submitted". In chat: "It's timing. Swap the sleep for a retrying assertion and move on." The engineer does exactly that, keeps 4 workers, and the two specs now read:

// submit-exam.spec.ts
import { test, expect } from '@playwright/test';

test.afterEach(async ({ request }) => {
  await request.post('/test/attempts/a-1/reopen');
});

test('submits the exam', async ({ page }) => {
  await page.goto('/exam/a-1');
  await page.getByRole('button', { name: 'Submit exam' }).click();
  await expect(page.getByRole('status')).toHaveText('Submitted');
});

On a run where submit-exam.spec.ts submits a-1 while the autosave test's save is in flight, what does the autosave test report, passed or failed, and what did its assertion receive?

// candidate-autosave.spec.tsimport { test, expect } from '@playwright/test'; test('shows Saved after typing', async ({ page }) => {  await page.goto('/exam/a-1');  await page.getByLabel('Answer to question 3').fill('Demand falls as price rises');  await expect(page.getByRole('status')).toHaveText('Saved');});

The 30-second version

A spec turned flaky when CI went from one worker to four. Name three suspects: a wait too short on a busier machine, data two tests share, a regression. Read what the failed assertion received. A browser context isolates cookies, not server rows, so give each worker its own data; then replace the sleep, and keep retries from hiding it.

When a suite turns flaky as workers go up, read what the assertion received, give each worker its own data, then fix the wait.

How it works

Step 1 of 12

Read the symptom. One spec fails 9 of 110 runs since workers went from one to four; reruns pass, and the release waits.

In the kitchen

four tables instead of one, to finish sooner → four workers instead of one (step 1)

Analogy: A duplicate bridge club's evening

The club sets up four tables instead of one to finish the evening sooner, and every table plays from the same boards.

At the bridge club	In the incident
four tables instead of one, to finish sooner	four workers instead of one (step 1)
every table's movement card naming Board 1	every spec using the one seeded attempt (step 6)
each player's own scorecard and pencil	each test's own browser context (step 6)
the caddy collecting Board 1 while another table is taking its hands out	the submit closing `a-1` during another test's save (steps 4–5)
a player who glances at the bidding tray once, after two minutes, before partner has bid	the sleep and the one-shot read (step 10)
boards numbered by table, 1 to 4	`parallelIndex` (step 8)
boards numbered by membership card, which can be any number	`workerIndex` (step 8)

Breaks down when: players at two tables would see the caddy take the board; two tests never see each other, only the server's refusal in a failure message.

Contrasting case: a table with its own board where a player still glanced before partner had bid. That is timing alone, and giving each worker its own data changes nothing.

In real terms: four workers run four tests at the same moment; whatever they share on the server, they share at that moment, and the failure message says which it was.

Examples, run for real

4.1 ExamHall EXH-5110: the suite as it stood, and the fix

The release's autosave test slept, then read the status once; submit-exam.spec.ts submitted the same seeded attempt a-1:

test('shows Saved after typing', async ({ page }) => {
  await page.goto('/exam/a-1');
  await page.getByLabel('Answer to question 3').fill('Demand falls as price rises');
  await page.waitForTimeout(1000);
  expect(await page.getByRole('status').textContent()).toBe('Saved');
});

The fix gives each worker its own attempt, keyed by test.info().parallelIndex (the seed creates a-0 to a-3), and replaces the last two lines with await expect(page.getByRole('status')).toHaveText('Saved'). Each line is one run:

4.1 release · 1 worker: passed
4.1 release · 4 workers, the submit during the save: failed, received "Already submitted"
4.1 release · 4 workers, busy machine: failed, received "Saving your answer…"
4.1 fix · 1 worker: passed
4.1 fix · 4 workers, the submit during the save: passed
4.1 fix · 4 workers, busy machine: passed

With one worker both causes vanish, which is why the serial run alone could not say which one was at work (step 3). Four workers showed two failures with two different Received values: the submit landing during the save, and a reply arriving after the sleep (step 4). The fix passes both (steps 9–10).

4.2 Chat MR !5410: one conversation per worker

A support-chat app's reply-draft.spec.ts and close-conversation.spec.ts shared the seeded conversation c-7, and CI ran with retries: 2. MR !5410 gives each worker its own conversation, c-${test.info().parallelIndex}:

4.2 before · 4 workers, retries 2, the close during the save on attempt 1: saves the reply draft flaky (failed, received "Conversation closed", then passed) · run exit 0
4.2 MR !5410 · 4 workers, retries 2, the close during the save on attempt 1: saves the reply draft passed on its first attempt · run exit 0

Before the MR the run was green and only the report's "flaky" label showed the collision; after it the test passes on its first attempt (steps 9, 11).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

The support-chat app's reply-draft test still sleeps and reads once:

It fails on some runs with 4 workers and never with 1. close-conversation.spec.ts closes the same seeded conversation, c-7, and reopens it afterwards. Two explanations are on the table: four workers make the machine busy, so the save's reply sometimes comes after the 1-second sleep; or the close-conversation spec, running at the same moment, closes c-7 during the save. In this lesson's model of the two, each acts on the first attempt of every 4-worker run and on nothing else: the busy machine holds that attempt's reply past the sleep, or the close lands during that attempt's save. A retry runs on a quiet machine, after c-7 is reopened. The CI job can run one more command, and its page shows only whether that command passed or failed.

Which command passes under one explanation and fails under the other?

// reply-draft.spec.tsimport { test, expect } from '@playwright/test'; test('saves the reply draft', async ({ page }) => {  await page.goto('/chat/c-7');  await page.getByLabel('Reply').fill('On it, checking your order now');  await page.waitForTimeout(1000);  expect(await page.getByRole('status').textContent()).toBe('Draft saved');});

Question 2 of 7

A streaming service's e2e suite runs with workers: 2. The seed creates two profiles, viewer-0 and viewer-1, one per worker, and a fixture picks the profile:

// player.spec.ts
import { test, expect } from './fixtures';

test('resumes at 12:30', async ({ page, viewer }) => {
  await page.goto(`/profiles/${viewer}`);
  expect(await page.getByRole('button', { name: /^Resume/ }).textContent()).toBe('Resume at 12:30');
});
test('saves a note', async ({ page, viewer }) => {
  await page.goto(`/profiles/${viewer}`);
  await page.getByLabel('Note').fill('watch with subtitles');
  await expect(page.getByRole('status')).toHaveText('Saved');
});
test('saves a second note', async ({ page, viewer }) => {
  await page.goto(`/profiles/${viewer}`);
  await page.getByLabel('Note').fill('skip the intro');
  await expect(page.getByRole('status')).toHaveText('Saved');
});

library.spec.ts holds two more tests of the same shape as 'saves a note'. Release 8.4 has a real bug: the resume button reads "Resume at 12:03", so 'resumes at 12:30' fails.

How many tests fail in the run?

// fixtures.tsimport { test as base } from '@playwright/test'; export const test = base.extend({  viewer: async ({}, use, testInfo) => {    await use(`viewer-${testInfo.workerIndex}`);  },});export { expect } from '@playwright/test';

Question 3 of 7

A ticketing site's hold-seat.spec.ts and release-hold.spec.ts both use the seeded seat 12, and on some 4-worker runs one changes the seat while the other is using it. A fix that seeds a seat per test is a week away. Until then, the team wants those two tests never to run at the same time, while the rest of the suite still runs four at a time.

Which change does that, according to Playwright's parallelism page?

Question 4 of 7

A payments dashboard's refund-note test sleeps, then reads once:

issue-refund.spec.ts issues the same seeded refund, r-31, and reopens it afterwards. CI runs 4 workers, and each suspect below is one of this lesson's lab models. After a red build the on-call engineer collects:

workers 4 · saves the agent note: failed · Expected: "Note saved" · Received: "Refund already issued
--repeat-each=5 --workers=1 · saves the agent note: 5 passed

Which suspect does this readout support?

// refund-note.spec.tsimport { test, expect } from '@playwright/test'; test('saves the agent note', async ({ page }) => {  await page.goto('/refunds/r-31');  await page.getByLabel('Agent note').fill('Customer sent the parcel back');  await page.waitForTimeout(1000);  expect(await page.getByRole('status').textContent()).toBe('Note saved');});

Question 5 of 7

In the interview you are given EXH-5110, a test that fails 8% of the time, and asked this; answer in at most 150 words:

How do you find out why this test fails?

Your answer (optional, stays on this page)

Question 6 of 7

A streaming service's e2e suite signs in once: a setup project signs in as the viewer pilot and saves the storage state, and every test starts from that file, with workers: 4. Two specs edit the same seeded playlist, p-9; when the server refuses a save, the page shows "Already deleted", and if a session ends, it shows "You have been signed out":

// delete-playlist.spec.ts
import { test, expect } from '@playwright/test';

test.afterEach(async ({ request }) => {
  await request.post('/test/playlists/p-9/reopen');
});

test('deletes the playlist', async ({ page }) => {
  await page.goto('/playlists/p-9');
  await page.getByRole('button', { name: 'Delete playlist' }).click();
  await expect(page.getByRole('status')).toHaveText('Deleted');
});

On a run where the delete lands while the rename's save is in flight, what does 'renames the playlist' report, passed or failed, and what did its assertion receive?

// rename-playlist.spec.tsimport { test, expect } from '@playwright/test'; test('renames the playlist', async ({ page }) => {  await page.goto('/playlists/p-9');  await page.getByLabel('Playlist name').fill('Sunday films');  await expect(page.getByRole('status')).toHaveText('Saved');});

Question 7 of 7

ExamHall's suite as it stood after t1's change: a retrying assertion, 4 workers, one seeded attempt a-1 for every spec. The fix seeds a-0 to a-3 and keys each spec's attempt by test.info().parallelIndex.

Which CI check fails on the suite with the shared attempt and passes on the fix, on a day the submit lands during the autosave's first attempt?

Go deeper

Pro tips

Repeat it at one worker, then at four, and read the Received value before touching the wait. --repeat-each=30 --workers=1, then --workers=4: if only the parallel run fails, look for shared data before timing (curriculum row's own tip), and let the failure's Received value decide between them. When: steps 3–4, before any fix. Status: EXECUTED.
Make a pass on retry visible the day you add retries. Retries keep the release train moving; a run that turns green on a retry hides the collision unless CI fails on it. When: step 11, while the data fix is in review. Status: EXECUTED.

Source

Playwright's behaviour can change between releases: the facts below are valid as of 2026-10-18 and were run on playwright 1.63.0.

Playwright, Parallelism: https://playwright.dev/docs/test-parallel (pw-test-parallel.txt), the curriculum row's first source. Quoted: "Playwright runs tests in separate worker processes, each with its own isolated BrowserContext , so cookies, storage and in-memory globals are already isolated. Flakiness comes from state that lives outside a single test." Supports steps 6–9.
Playwright, Best Practices: https://playwright.dev/docs/best-practices (pw-best-practices.txt), the row's second source. Supports steps 6 and 10.
Playwright, Authentication: https://playwright.dev/docs/auth (pw-auth.txt), the row's third source: a setup project that saves one sign-in for every test. Supports step 6.
Playwright, Command line: https://playwright.dev/docs/test-cli (pw-test-cli.txt). Supports steps 11–12.
Playwright, TestInfo: https://playwright.dev/docs/api/class-testinfo (pw-api-testinfo.txt). Quoted: "It is guaranteed that workers running at the same time have a different parallelIndex ." Supports step 8.

Checked against the tools and docs as of 2026-10-18.

Predict first