I once watched a Node.js import job climb to 6GB of memory usage and die, on a file that was 4GB on disk. The bug wasn't a memory leak in the traditional sense. It was a readable stream reading a CSV faster than a slow database write could keep up, with no backpressure handling between them. Every unconsumed chunk sat in memory until there wasn't any left. Fixing it took one function call.
TL;DR: A Node.js stream is readable, writable, or both (transform/duplex). Backpressure happens when a fast producer overwhelms a slow consumer, and unhandled backpressure is the most common cause of memory blowups in stream-based code. Use
pipeline()fromnode:stream/promisesinstead of.pipe(), it handles backpressure and error propagation for you, and it's the version you should default to in 2026.
The Four Stream Types, Briefly
Node.js streams come in four flavors, and knowing which one you're working with tells you what methods are available:
| Type | Direction | Example |
|---|---|---|
| Readable | Source you read from | fs.createReadStream(), an HTTP response body |
| Writable | Destination you write to | fs.createWriteStream(), an HTTP request body |
| Duplex | Both readable and writable, independently | A TCP socket |
| Transform | Duplex where output is derived from input | zlib.createGzip(), a CSV parser |
If you've only ever used fs.readFile(), you've been avoiding streams entirely, loading the whole file into memory at once. That's fine for a 10KB config file. It's not fine for a 4GB CSV.
Backpressure: The Part Tutorials Skip
Here's the mechanism. Every writable stream has an internal buffer with a size limit, highWaterMark, defaulting to 16KB for most streams. When you call .write() and the buffer is full, .write() returns false. That's the writable stream telling you: stop sending data until I emit a 'drain' event.
If you ignore that signal and keep writing anyway, Node.js will still accept the data, it just queues it in memory without bound. That's exactly what happened in my CSV import: the read side kept producing chunks faster than the database write side (with its network round-trips) could consume them, and nothing was pausing the producer.
// Manual backpressure handling, this is what pipe()/pipeline() do for you
function copyManually(readable, writable) {
readable.on('data', chunk => {
const canContinue = writable.write(chunk);
if (!canContinue) {
readable.pause(); // stop reading until the writable catches up
}
});
writable.on('drain', () => {
readable.resume(); // writable caught up, resume reading
});
readable.on('end', () => writable.end());
}You almost never write this by hand. .pipe() and pipeline() implement exactly this pause/resume dance internally. But understanding it is what makes stream bugs debuggable instead of mysterious.
pipe() vs pipeline(): Use pipeline()
.pipe() has been in Node.js since the beginning, and it still works. The problem is error handling: if the source stream errors, .pipe() does not automatically destroy the destination stream. You end up manually wiring .on('error', ...) handlers on every stream in the chain, and it's easy to miss one, leaving a file handle or socket open.
pipeline(), available as a promise-based API from node:stream/promises since Node 15, fixes this. One error anywhere in the chain destroys every stream in the chain and rejects a single promise.
import { pipeline } from 'node:stream/promises';
import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
async function compressFile(inputPath: string, outputPath: string): Promise<void> {
await pipeline(
createReadStream(inputPath),
createGzip(),
createWriteStream(outputPath),
);
}Three lines, and pipeline() handles backpressure between all three streams, cleans up on error, and gives you a real promise to await. That's a .gzip file compression utility with correct memory behavior, not just correct output.
Building a Transform Stream: CSV Row Processing
The case where streams actually earn their complexity is processing large files row by row without loading them into memory. Here's a transform stream that parses CSV lines and filters them, the exact shape of the fix I applied to that 4GB import:
import { Transform } from 'node:stream';
class CsvRowFilter extends Transform {
#buffer = '';
constructor(private predicate: (row: string[]) => boolean) {
super({ objectMode: false });
}
_transform(chunk: Buffer, _encoding: string, callback: (error?: Error | null) => void) {
this.#buffer += chunk.toString('utf8');
const lines = this.#buffer.split('\n');
this.#buffer = lines.pop() ?? ''; // keep the incomplete last line
for (const line of lines) {
const row = line.split(',');
if (this.predicate(row)) {
this.push(row.join(',') + '\n');
}
}
callback();
}
_flush(callback: (error?: Error | null) => void) {
if (this.#buffer) {
const row = this.#buffer.split(',');
if (this.predicate(row)) {this.push(row.join(',') + '\n');}
}
callback();
}
}This processes the file one chunk at a time (typically 64KB), regardless of whether the file is 4MB or 4GB. Memory usage stays flat because nothing accumulates beyond the current chunk and the small leftover buffer for split lines.
Have you ever tried to JSON.parse() a multi-gigabyte file and watched the process hang? That's the same problem, solved the same way, streaming JSON parsers use a transform stream instead of reading the whole thing into a string first.
Interop with Web Streams
Since Node 17, you can convert between Node streams and the Web Streams API that fetch() uses, which matters if you're piping a file to an outgoing HTTP request:
import { Readable } from 'node:stream';
import { createReadStream } from 'node:fs';
const nodeStream = createReadStream('./large-file.bin');
const webStream = Readable.toWeb(nodeStream);
await fetch('https://api.example.com/upload', {
method: 'POST',
body: webStream,
duplex: 'half', // required by fetch() when streaming a request body
});That duplex: 'half' option isn't optional, fetch() throws without it when the body is a stream. It's an easy one-line bug to miss the first time you try this.
When Streams Are Overkill
Not every file operation needs a stream. If you're reading a config file, a small JSON payload, or anything reliably under a few megabytes, fs.readFile() is simpler and the memory cost is trivial. Reach for streams when the input size is unbounded, unknown ahead of time, or large enough that loading it whole would be a real memory concern, roughly anything you'd hesitate to load fully into a browser tab.
Conclusion
Streams solve exactly one problem: processing data that's too large, or arrives too slowly, to load all at once. Backpressure is the mechanism that keeps that processing from blowing up memory, and pipeline() is the API that handles it correctly without you writing the pause/resume logic by hand. Default to pipeline() over .pipe() in anything you write today, and reach for a custom Transform stream only when you need to process data mid-flight, not just move it from A to B.