Chapter 29: Node.js Streams
1. What is a stream? (the simplest mental model)
A stream is a continuous flow of data that you can read from or write to piece by piece — instead of loading everything into memory at once.
Think of it like a water pipe:
- You don’t wait for the whole river to arrive before you start using water
- You use it as it flows
- You can connect pipes → send water from one pipe to another
Four main types of streams in Node.js
| Type | What it does | Readable? | Writable? | Classic real-world example |
|---|---|---|---|---|
| Readable | You read data from it | Yes | No | Reading a large file, HTTP response, stdin |
| Writable | You write data to it | No | Yes | Writing to a file, HTTP request body, stdout |
| Duplex | Can read and write (both directions) | Yes | Yes | TCP socket, WebSocket connection |
| Transform | A special Duplex — reads, modifies, writes | Yes | Yes | gzip compression, JSON parsing, uppercase text |
The golden rule everyone must remember:
Streams exist so you never load huge data entirely into memory They process data in small chunks (usually ~64 KB by default)
This is why Node.js can handle gigabyte-sized files or video streaming with very low memory usage.
2. The classic “don’t do this” example (very important)
Wrong way — loads entire file into memory
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 |
import fs from 'node:fs/promises'; async function copyFileBad() { const content = await fs.readFile('big-video.mp4'); // → 4 GB in RAM! await fs.writeFile('copy.mp4', content); // → another 4 GB } |
→ If file is 10 GB → your server dies (out of memory)
Right way — using streams (memory usage stays almost constant)
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import fs from 'node:fs'; function copyFileGood() { const readStream = fs.createReadStream('big-video.mp4'); const writeStream = fs.createWriteStream('copy.mp4'); readStream.pipe(writeStream); writeStream.on('finish', () => { console.log('Copy finished successfully'); }); } |
→ Memory usage ≈ 64 KB no matter how big the file is
3. All four types with real, runnable examples
3.1 Readable Stream – reading data chunk by chunk
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
import fs from 'node:fs'; const readStream = fs.createReadStream('large-log.txt', { encoding: 'utf-8', highWaterMark: 1024 * 64 // 64 KB chunks (default) }); readStream.on('data', (chunk) => { console.log(`Received {chunk.length} bytes`); console.log(`Chunk preview: ${chunk.slice(0, 50)}...`); }); readStream.on('end', () => { console.log('Finished reading entire file'); }); readStream.on('error', (err) => { console.error('Read error:', err.message); }); |
Real-world use case: Reading CSV / JSON lines / log files too large for memory
3.2 Writable Stream – sending data chunk by chunk
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
import fs from 'node:fs'; const writeStream = fs.createWriteStream('output.txt'); for (let i = 0; i < 100000; i++) { writeStream.write(`Line ${i}: Hello from stream\n`); } writeStream.end(); // very important — signals end of writing writeStream.on('finish', () => { console.log('All data written successfully'); }); |
Important: .write() returns false when the internal buffer is full → you should wait for ‘drain’
JavaScript
|
0 1 2 3 4 5 6 7 8 |
writeStream.on('drain', () => { console.log('Buffer drained → safe to write more'); }); |
3.3 Piping – the most beautiful feature of streams
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
import fs from 'node:fs'; import zlib from 'node:zlib'; // Read → gzip compress → write compressed file fs.createReadStream('big-file.txt') .pipe(zlib.createGzip()) .pipe(fs.createWriteStream('big-file.txt.gz')); console.log('Compression started...'); |
Multiple pipes (very common)
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import fs from 'node:fs'; import { Transform } from 'node:stream'; const upperCase = new Transform({ transform(chunk, encoding, callback) { callback(null, chunk.toString().toUpperCase()); } }); fs.createReadStream('input.txt') .pipe(upperCase) .pipe(fs.createWriteStream('output-upper.txt')); |
3.4 Duplex & Transform – the most powerful ones
Transform example – real use-case
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
import { Transform } from 'node:stream'; const replaceWord = new Transform({ transform(chunk, encoding, callback) { const transformed = chunk.toString().replace(/hello/gi, 'HI'); callback(null, transformed); } }); process.stdin .pipe(replaceWord) .pipe(process.stdout); |
Run → type text → every “hello” becomes “HI”
Very common real-world transform examples:
- Gzip / gunzip
- JSON parsing line-by-line (JSONStream, ndjson)
- CSV parsing (csv-parser)
- Encryption / decryption
- Image resizing on-the-fly
- Replacing text / sanitizing HTML
5. Modern best practices & patterns (2025–2026)
Pattern 1 – Pipe + error handling (very important)
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
import fs from 'node:fs'; const rs = fs.createReadStream('input.txt'); const ws = fs.createWriteStream('output.txt'); rs.pipe(ws); rs.on('error', err => console.error('Read error:', err)); ws.on('error', err => console.error('Write error:', err)); ws.on('finish', () => console.log('Done')); |
Pattern 2 – Promise wrapper for streams (very clean)
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
function streamToPromise(stream) { return new Promise((resolve, reject) => { stream.on('error', reject); stream.on('end', resolve); stream.on('finish', resolve); }); } // Usage await streamToPromise( fs.createReadStream('input.txt').pipe(fs.createWriteStream('output.txt')) ); |
Pattern 3 – HTTP file download using streams
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
import { createServer } from 'node:http'; import fs from 'node:fs'; createServer((req, res) => { const filePath = 'large-video.mp4'; const stat = fs.statSync(filePath); const fileSize = stat.size; res.writeHead(200, { 'Content-Type': 'video/mp4', 'Content-Length': fileSize }); fs.createReadStream(filePath).pipe(res); }).listen(3000); |
→ Browser can start playing video before entire file is read
6. Common mistakes & how to avoid them
| Mistake | Consequence | Fix |
|---|---|---|
| Not handling ‘error’ on streams | Silent failures, memory leaks | Always add .on(‘error’, …) |
| Not calling .end() on writable stream | Client hangs forever | Always call .end() when done writing |
| Using .pipe() without error handling | Errors disappear | Use pipeline() from stream/promises |
| Loading entire stream into memory | Out of memory on large files | Use .pipe() or process chunks |
| Forgetting to resume paused stream | Data loss / hanging | Understand backpressure & ‘drain’ |
Modern helper (highly recommended)
JavaScript
|
0 1 2 3 4 5 6 7 8 9 10 11 12 |
import { pipeline } from 'node:stream/promises'; await pipeline( fs.createReadStream('input.txt'), zlib.createGzip(), fs.createWriteStream('output.txt.gz') ); |
→ Automatically handles errors & cleanup
Summary – Quick decision guide
| You want to… | Use this type / pattern | Typical real-world example |
|---|---|---|
| Read large file / response | Readable stream + .pipe() | File download, log tailing, HTTP streaming |
| Write large amount of data | Writable stream + .write() + .end() | Generating CSV, logging, uploading to S3 |
| Transform data on-the-fly | Transform stream | Compression, encryption, text replacement |
| Connect streams safely | stream/promises.pipeline() | Modern, promise-based, error-safe piping |
| Handle backpressure | Listen for ‘drain’ event | Writing very fast to slow destination |
Would you like to go much deeper into any part?
- Backpressure – how to handle it properly
- pipeline() vs manual .pipe() – when & why
- Object mode streams (very powerful)
- Real production example – file upload + resize + S3 stream
- Stream error handling patterns in Express/Fastify
Just tell me which direction feels most useful — I’ll continue with detailed, production-ready code examples. 😊
