Streams and backpressure

Each stage buffers up to its own mark; pipe() leaves a failure half-open, pipeline() tears the whole chain down.

Before you read: predict

ExamHall's results export streams a Postgres cursor through a CSV transform into S3:

upload is a Writable that sends each part to S3. It completes the multipart upload in its final step, once it has been ended; destroying it aborts the upload. At row 300 the database connection drops, and cursor is destroyed with an error.

Once nothing else happens, what state is upload in?

cursor.pipe(toCsv).pipe(upload);cursor.on('error', log);

The 30-second version

Every stream buffers up to its highWaterMark: by default 64 KiB of bytes, or 16 objects in object mode. A stalled chain fills each stage in turn, then stops reading its source. pipe() honours that but passes no failure on: the stream that failed is destroyed, the others are left open. pipeline() destroys them all.

Each stage buffers up to its own mark; pipe() leaves a failure half-open, pipeline() tears the whole chain down.

How it works

Step 1 of 10

Two queues. A Readable holds data not yet read (readableLength); a Writable, data not yet written (writableLength). A Transform has both.

In the kitchen

each station's short waiting lane, ending at a painted line → a stage's buffer and its highWaterMark (steps 1–2)

Analogy: An automatic tunnel car wash (vehicle services)

Cars roll through soap, brushes and a dryer, one station after another.

At the car wash	In the stream
each station's short waiting lane, ending at a painted line	a stage's buffer and its `highWaterMark` (steps 1–2)
a coach longer than the lane still drives in whole; it just fills the lane past the line	a threshold, not a cap (step 3)
with the dryer stuck, lanes fill one by one until the entrance gate stops taking cars	a stalled chain stops reading its source (step 7)
the brushes break and only their own attendant is told: cars sit soapy in the lanes, and the exit barrier never lifts	`pipe()` after an error (step 5)
the master stop button halts every station, opens every lane and tells the cashier once	`pipeline()` (step 8)
a driver who leaves the queue halfway loses the place; it is not kept for later	leaving `for await` early (step 10)

In real terms: memory is bounded by the marks you set, stage by stage; cleanup after a failure happens only if something tears the whole chain down.

Breaks down when: a car can reverse out and rejoin. A destroyed stream never yields again.

Contrasting case: an array built with await on every row. There is no buffer to fill and no source to pause: memory grows with the data, which is what streams avoid.

Examples, run for real

4.1 ExamHall EXH-3700: the export that kept its connection

ExamHall's export ran cursor.pipe(toCsv).pipe(gzip).pipe(upload), with a real gzip stream. S3 failed the third part (EXH-3700). MR !4000 wired the same four streams with one call:

pipeline(cursor, toCsv, gzip, upload, (err) => {
  if (err) log('export failed', err);
});

EXH-3700 pipe():     cursor open · toCsv open · gzip open · upload destroyed · cursor paused true · pool connections in use 1
MR !4000 pipeline(): cursor destroyed · toCsv destroyed · gzip destroyed · upload destroyed · callback 1x: S3 part failed · pool connections in use 0

With pipe(), the failed upload destroyed only itself (step 5); the paused cursor kept its pool connection until the process ended. pipeline() destroyed all four, released the connection (step 9) and reported the one error (step 8).

4.2 Payments: where a stall's data waits

A payments ledger's 1,000 entries are piped into an object-mode upload with highWaterMark: 100, and S3 stalls on the first entry:

ledger.pipe(upload), upload stalled: upload.writableLength 100 · ledger.readableLength 16 · entries read from the source 116 of 1,000

The upload's queue holds 100 entries, its mark; the ledger stream then holds 16, its own default mark, and stops reading (steps 2 and 7). The other 884 entries stay in the database.

Picture it

Concept illustration for this lesson. — An illustration to help it stick. The mechanism is in the sections above.

Try it

Answer first, then check. Nothing is saved.

Question 1 of 7

A chat app's history export first shows the conversation's first 50 messages as a preview, then exports the whole conversation from the same stream. history.stream() returns a Readable of the conversation's 1,000 messages, oldest first; pipeline comes from node:stream/promises; upload records the CSV lines it receives.

What does the export do?

const messages = history.stream();const preview = [];for await (const m of messages) {  preview.push(m);  if (preview.length === 50) break;}await pipeline(messages, toCsv, upload);

Question 2 of 7

A payments ledger exports its 100,000 rows with rows.pipe(toCsv).pipe(upload). rows is an object-mode Readable that reads the rows from the database, with highWaterMark: 16. toCsv turns each row into one 200-byte CSV line; its writable side is in object mode and its readable side carries bytes, both with the default highWaterMark. upload is a byte Writable with the default highWaterMark. S3 stalls on the very first part: upload's write callback is never called. The process runs Node 24 on Linux.

Once nothing moves, about how many rows have been read from the database?

Question 3 of 7

The interviewer says: "Our ledger export must move millions of rows to a slow destination. Write exportLedger(rows, dest): rows is an async iterable or a Readable of { id, amount }, dest a byte Writable. Write one CSV line id,amount per row, in order. Return a promise that resolves once dest has finished. Memory must stay flat however slow dest is, and on any failure the promise rejects with that error and nothing is left open." You have 15 minutes and a plain editor.

Say your test list first, then write it.

Your answer (optional, stays on this page)

Question 4 of 7

A box office's nightly report writes the day's 20 orders to a file:

toCsv calls its callback with an error for an order that has no seat, and order 7 has none. No other listener is attached.

What happens?

orders.pipe(toCsv).pipe(file);file.on('error', alert);

Question 5 of 7

A video platform sends a film's encoded segments to a viewer:

encoder.next() returns the film's next 40 KiB segment (40,960 bytes); the film has 2,000 of them. segments has the default highWaterMark, and the process runs Node 24 on Linux. The viewer's connection stalls on the first segment: viewer.send() never settles.

Once nothing moves, what is segments.readableLength?

const segments = new Readable({  read() { this.push(encoder.next()); },});for await (const seg of segments) {  await viewer.send(seg);}

Question 6 of 7

A video platform's download route sends a recording with file.pipe(meter).pipe(res): file is a read stream over the recording, meter a Transform that counts the bytes sent for the viewer's data plan, res the HTTP response, and pipeline and finished come from node:stream. Viewers who close the tab mid-download leave the recording's file handle open, and after a busy day the server runs out of file descriptors.

Which replacement closes the file handle when a viewer leaves?

Question 7 of 7

A payments ledger imports a bank's settlement entries. The bank may refuse a single entry, so each post has its own try/catch:

entries is a Readable of 600 entries, and no 'error' listener is attached to it. While entry 3 is being posted, the bank's connection resets and entries is destroyed with new Error('ECONNRESET'); the post of entry 3 itself completes.

What is logged?

let posted = 0;async function importAll() {  for await (const entry of entries) {    try {      await post(entry);      posted++;    } catch (err) {      log(`skipped one: ${err.message}`);    }  }  log(`done: ${posted}`);}importAll().catch((err) => log(`failed after ${posted}: ${err.message}`));

Go deeper

Pro tips

Set highWaterMark on object streams that carry big objects. The default counts objects, not bytes, so 16 large rows can sit in one buffer. When: streaming wide rows, images or documents in object mode (step 2). Verify (RAN, tips.mjs):
```
tip 1 object mode, highWaterMark default: readableLength 16 objects · 16 MB held
tip 1 object mode, highWaterMark 2: readableLength 2 objects · 2 MB held
```
Await finished(stream) from node:stream/promises to learn when one stream is done or failed. It settles on 'finish', 'end', an error or an early close, so you need not wire each event yourself. When: a handler must wait for one stream it did not build into a pipeline (step 8). Verify (RAN, tips.mjs):
```
tip 2 finished(): ended normally resolved · write error rejected disk full · destroyed early rejected ERR_STREAM_PREMATURE_CLOSE
```

Source

Quotes are checked verbatim by quote-check.mjs. Every graded behaviour is executed, so no item is keyed on these pages; they back the teaching. Valid as of 2026-10-12.

Node.js, Backpressuring in Streams (the curriculum row's first source), https://nodejs.org/en/learn/modules/backpressuring-in-streams (node-backpressure.txt). On a .pipe() chain with no error handlers: "If a chunk of data were to fail to be properly received, the Readable source or gzip stream will not be destroyed." (step 5); pipeline is "a module method to pipe between streams forwarding errors and properly cleaning up and providing a callback when the pipeline is complete." (step 8).
Node.js v24 API, Stream (the row's second source, its stream.pipeline() section), https://nodejs.org/docs/latest-v24.x/api/stream.html (node-stream-api.txt). "stream.pipeline() will call stream.destroy(err) on all streams except:" streams that already ended or finished (step 8); for readable.pipe(): "if the Readable stream emits an error during processing, the Writable destination is not closed automatically" (step 5); getDefaultHighWaterMark: "Defaults to 16 for objectMode. For byte streams, it defaults to 65536 (64 KiB) on non-Windows platforms and 16384 (16 KiB) on Windows." (step 2); writableLength "contains the number of bytes (or objects) in the queue ready to be written" (step 1); async iteration: "If the loop terminates with a break, return, or a throw, the stream will be destroyed." (step 10).

Checked against the tools and docs as of 2026-10-12.

Predict first