WittCode💻

Node Streams

By

Learn what a Node stream is including readable, writable, duplex, and transform streams. We will also go into why Node streams are a good idea for working with large amounts of data.

Table of Contents 📖

What is a Stream?

A stream is a collection of data where the data is available chunk by chunk, or piece of data by piece of data. This makes them handy for working with large amounts of data as all the data isn't loaded into memory at once. For example, instead of sending a whole video file to a client from a server, we can create a stream to the video and send down chunks of it. Video files are often several GBs in size, so sending a whole video file at once would spike a server's memory and slow it down. Streams allow the video file to be sent in chunks, saving the server's performance.

Types of Streams

Node has four types of streams: readable, writable, duplex, and transform.

  • Readable - Used to read data from a source, such as reading from a file.
  • Writable - Used to write data to a destination, such as writing to a file.
  • Duplex - Used to read and write data, such as a TCP socket connection.
  • Transform - Used to transform data, such as compressing or decompressing data.

All of these streams work by using events. For example, readable streams emit events such as data (chunk of data is available to the consumer) and end (no more data is available to be consumed) while writable streams emit events such as drain (no more data can be written to the stream) and finish (all data has been written to the stream).

Writable Streams

Lets first focus on a writable stream by writing a large amount of data to a file.

import fs from 'fs';

const fileWriteStream = fs.createWriteStream('./largeFile.txt');

for (let i = 0; i < 1000000; i++) {
  fileWriteStream.write('Download WittCepter the greatest chrome extension in the world! It is amazing omg!!!!\n');
}

fileWriteStream.end();
  • Import the Node file system module. The fs module provides methods for creating both read and write streams from and to files
  • Create a write stream to write data to a file
  • Write 1,000,000 lines of data to the file. Note that we are writing each line as a separate chunk of data as opposed to everything at once
  • End the stream

Now lets check the size of this large file.

const {size} = fs.statSync('./largeFile.txt');
const mb = size / (1024*1024);
// The file is 82.96966552734375 bytes in size
console.log('The file is ' + mb + ' bytes in size');

Now, imagine sending this 83 MB file down the wire from a server to a client. This can take a decent amount of time, even worse so if the connection is poor.

Readable Streams

Instead of sending this 83 MB file down the wire, lets send it in chunks using a readable stream.

import fs from 'fs';
import http from 'http';

const server = http.createServer();

server.on('request', (req, res) => {
  const readableStream = fs.createReadStream('./largeFile.txt');
  readableStream.pipe(res);
});

server.listen(5001);
  • Import the http and fs libraries.
  • Create a bare bones HTTP server.
  • Set up an event listener for requests.
  • When a request is received, create a readable stream and pipe it to the response. The pipe method is very useful for working with streams. It takes a readable stream and passes it to a writable stream. Here it reads data from the large file and pipes it to the response object, a writeable stream that sends data to the client.
  • Listen on port 5001

Now lets contact the server.

curl localhost:5001
...
Download WittCepter the greatest chrome extension in the world! It is amazing omg!!!!
Download WittCepter the greatest chrome extension in the world! It is amazing omg!!!!
Download WittCepter the greatest chrome extension in the world! It is amazing omg!!!!

Working with the large data this way prevents the memory of the server from spiking.

Duplex Streams

As stated previously, duplex streams are streams that can both read and write. An example duplex stream is the Node net.Socket object.

import net from 'net';

// Server
const server = net.createServer(socket => {
  socket.on('data', data => {
    console.log(data.toString());
    socket.write('Hello from server!');
  });
});
server.listen(1235);

// Client
const socket = net.connect({
  port: 1235
});
socket.on('connect', () => {
  socket.write('Hello from client!');
});
socket.on('data', data => {
  console.log(data.toString());
});

Here the server establishes a socket connection with the client when it connects. Both the server and client then read and write data to this socket. For example, socket.write sends data while socket.on listends for data.

Transform Streams

A transform stream is a stream that transforms data from one form to another. For example, we can use a transform stream to uppercase everything in our large file.

import fs from 'fs';
import {Transform} from 'stream';

const uppercaseStream = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase());
  }
});

fs.createReadStream('./largeFile.txt')
  .pipe(uppercaseStream)
  .pipe(fs.createWriteStream('./largeFileUppercase.txt'));
  • Import the fs module and the Transform object from the stream module. The stream module provides a way of handling stream data. The Transform object allows us to create a transform stream.
  • Create a transform stream to uppercase the data it receives. To do this we simply implement the transform method. This method accepts 3 arguments: the chunk (buffer to be transformed), the encoding, and a callback function.
  • Call the callback function to signal success or failure. The first argument is null as there is no error, the second argument is the data that has been transformed. If something went wrong, we would pass an error object as the first argument of the callback.
  • Create a read stream from our large text file and pipe it to our uppercase stream.
  • Create a write stream to write our transformed data to a new file.

If we view the contents of the largeFileUppercase.txt file, it should be all uppercase.

cat largeFileUppercase.txt
...
...
DOWNLOAD WITTCEPTER THE GREATEST CHROME EXTENSION IN THE WORLD! IT IS AMAZING OMG!!!!
DOWNLOAD WITTCEPTER THE GREATEST CHROME EXTENSION IN THE WORLD! IT IS AMAZING OMG!!!!