Skip to content

Delegate by checkability

Hand the assistant work in proportion to how cheaply you can check it, and name the check that would prove it wrong.

Predict first

Before you read: predict

ExamHall's results team has four tasks this sprint. Each comes with the check it would get today before it merges.

Which one would you hand to the assistant?

Pick one

The 30-second version

Generated output saves time only once you know it is right. Before accepting, name the check a wrong version would fail. Mappings, scaffolds and drafts show their mistakes in one run. Security, concurrency, migrations and version-specific code fail only under another caller, concurrent requests, a later time, production data or another library version, unless a test creates that condition.

Hand the assistant work in proportion to how cheaply you can check it, and name the check that would prove it wrong.

How it works

Step 1 of 12

Checking is the cost. A generated output saves time only after you know whether it is right; what that check costs decides its worth.

Analogy: A bathroom fitter deciding which jobs to hand to an apprentice

She gives the apprentice the wall tiling: a crooked row or the wrong grout colour shows the moment she looks at the wall. She keeps the waterproof membrane under the shower tray. A membrane that will leak looks exactly like one that will hold, and it fails only when water runs for days behind finished tiles. A flood test, the tray plugged and filled to a marked line before any tile goes on, fails a leaking membrane; with that test, the membrane can go to the apprentice too.

In the bathroom With a generated output
Looking at the tiled wall A check that shows a mistake in one run (step 3)
A membrane that looks the same, leaking or not Code that fails only under a condition (step 4)
Water running for days behind the tiles Another caller, concurrent requests, production data
The flood test before tiling A test that creates the condition and fails a wrong copy (step 6)
"What would show me this is wrong?" Name the check (step 2)
Her sense that the apprentice saved the afternoon A self-reported speed-up (steps 8–9)
The start and finish times on the job sheets Measured times on comparable tasks (steps 10, 12)

Breaks down when: a flood test runs once and the tray either holds or leaks, while concurrent requests may never overlap in a test run; the test must force the overlap, not hope for it (step 6). And an apprentice learns from each job, while an assistant's next draft carries no memory of the last correction.

Contrasting case: deciding what to build, the scope and the design, is not work to delegate by checkability at all: those calls stay yours (aiassist-failure-modes step 8).

In real terms: before handing work over, ask which check a wrong output would fail; hand over what such a check covers, and write the check first for the rest.

Examples, run for real

4.1 EXH-2470: an admin-only guard and an admin-only suite

ExamHall's admin console lets an admin re-open a closed exam. The assistant wrote the guard:

// reopen.mjs (generated)
export function reopenExam(user, exam) {
  if (user.role !== 'admin') return { status: 403 };
  return { status: 200, exam: { ...exam, state: 'open' } };
}

Then it adds one test, MR !2770's: a proctor asks to re-open, and expects 403.

4.1 EXH-2470 existing tests (18, each as an admin) · generated guard: 18 of 18 pass · copy with the role check deleted: 18 of 18 pass
4.1 MR !2770 adds "a proctor is refused" · generated guard: 19 of 19 pass · copy with the role check deleted: 18 of 19 pass, failing: a proctor is refused

Every existing test calls as an admin, so the suite passes the copy that lets anyone re-open (steps 4–5). The added test creates the missing condition, another caller, and only the copy fails it (step 6).

4.2 Payments ledger: currency minor units, checked in one run

The ledger stores amounts in minor units, so each currency needs its number of decimal places. The assistant drafted the table; one run compares it with the currency data in the project's Node.

4.2 assistant's table: GBP 2 · EUR 2 · USD 2 · JPY 2 · KWD 2 · CHF 2
4.2 Intl.NumberFormat on node 24.19.0: GBP 2 · EUR 2 · USD 2 · JPY 0 · KWD 3 · CHF 2
4.2 rows that differ: 2 of 6 (JPY, KWD)

The run names the two wrong rows at once. A draft whose mistakes show against known values in one run is cheap to check (step 3).

Try it

Answer first, then check. Nothing is saved.

Question 1 of 5

The assistant wrote a migration for ExamHall's answers table, which every autosave writes to (about 40 M rows in production):

-- 0042_answers_attempt_idx.sql
CREATE INDEX answers_attempt_idx ON answers (attempt_id);

The release goes out while exams are in progress.

Which one check, run before merge, can show this migration unsafe for that release?

Pick one

Question 2 of 5

A payments-ledger team has had an AI assistant for a quarter. Before renewing it, the lead sends a survey asking how much time the assistant saves each developer. The median answer is "about a fifth of my time".

As METR's July 2025 study reports developers' own estimates, what does that median show?

Pick one

Question 3 of 5

An interviewer asks: "How would you find out whether the assistant really speeds up your team on the payments ledger?" Which plan gathers the kind of evidence METR's July 2025 study relied on?

Pick one

Question 4 of 5

A ticketing team will let the assistant rewrite holdSeat for speed, and will accept a rewrite only if their test would fail one that can sell a seat twice. Today's function and test:

// holdSeat.test.mjs
it('sells the last seat once', async () => {
  const seats = fakeSeats({ e1: 1 });
  expect(await holdSeat(seats, 'e1')).toBe('held');
  expect(await holdSeat(seats, 'e1')).toBe('sold out');
});

The test passes on today's function, and also on a copy with the seats.lock line and its release() deleted.

Which change to the test makes it fail that copy, while it still passes today's function?

Code for this question
// holdSeat.mjsexport async function holdSeat(seats, eventId) {  const release = await seats.lock(eventId);  try {    const left = await seats.available(eventId);    if (left < 1) return 'sold out';    await seats.setAvailable(eventId, left - 1);    return 'held';  } finally { release(); }}
Pick one

Question 5 of 5

In a hiring-manager round for a payments team, you say you check the assistant's speed-up on your own tasks. The interviewer replies: "METR found AI makes experienced developers 19% slower.

Why use it at all?" As METR's study page reads as of 2026-10-02, which reply is accurate?

Pick one

Go deeper

Pro tips
  1. Before accepting a suggestion, name the check that would prove it wrong, and run it; if you cannot name one, you are trusting, not verifying. (The curriculum row's tip.) When: every generated change, and first of all in security, concurrency and migration code (steps 2, 4). Verify: the named check for the re-open guard, "a proctor is refused", fails the copy without the role check and passes the guard. Status: RAN.
Source
  • S1 · METR (Joel Becker, Nate Rush, Beth Barnes, David Rein), Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 10 July 2025: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/, read 2026-10-02, saved by fetch-sources.mjs as sources/metr-2025-study.txt. The curriculum row's source. The 16-developer randomised trial: "they take 19% longer"; "they still believed AI had sped them up by 20%"; the method ("we randomly assign each issue"); the limits it lists ("We do not claim that our developers or repositories represent a majority or plurality of software development work"); and, as of 2026-10-02, a notice that "these historical results no longer reflect the current impact of AI models on open-source developer productivity".
  • S2 · PostgreSQL 18 documentation, CREATE INDEX: https://www.postgresql.org/docs/current/sql-createindex.html, read 2026-10-02, saved as sources/pg-create-index.txt. The curriculum row's source: "a standard index build locks out writes (but not reads) on the table until it's done". The run is on PostgreSQL 17, whose page says the same.
  • S3 · METR, We are Changing our Developer Productivity Experiment Design, 24 February 2026: https://metr.org/blog/2026-02-24-uplift-update/, read 2026-10-02, saved as sources/metr-2026-uplift-update.txt. Linked from S1's update line. Teaching only (step 11): "we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools", because developers chose "not to submit some tasks because they did not want to do them without AI". Keys nothing.
  • S4 · PostgreSQL 18 documentation, 13.3 Explicit Locking: https://www.postgresql.org/docs/current/explicit-locking.html, read 2026-10-02, saved as sources/pg-explicit-locking.txt. Teaching only: INSERT takes ROW EXCLUSIVE, which "Conflicts with the SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, and ACCESS EXCLUSIVE lock modes."
  • S5 · PostgreSQL 18 documentation, pg_locks: https://www.postgresql.org/docs/current/view-pg-locks.html, read 2026-10-02, saved as sources/pg-view-pg-locks.txt. Teaching only: "False indicates that this process is currently waiting to acquire this lock".

Checked against the tools and docs as of 2026-10-02.