Compression and Decompression of Data Using Zlib in Node.js

What is data compression?

When working with large systems or moving large amounts of data around, it is helpful to be able to compress/decompress the data. Node.js provides an excellent library in the Zlib module that allows you to compress and decompress data in buffers easily and efficiently. 

You need to keep in mind that compressing data takes CPU cycles, so you should be certain of the benefits of compressing data before you incur the compression/decompression cost. Zlib supports these compression methods: 

• gzip/gunzip: Standard gzip compression. 
• deflate/inflate: Standard deflate compression algorithm 
• Brotli spec 

Why do we need to compress data?

Compression in computer terms means reducing the physical size of data such that it occupies less storage space and memory, Compressed files are, therefore, easier to transfer because there is a sizable amount of reduction in the size of data to be transferred. 

This results in a reduction in the time needed for file transfer as well as a reduction in the bandwidth utilization even for a slow network. 

Thus, the process of reducing the amount of data required to represent information is called compression. 

There are two types of compression: lossless and lossy.

Lossless compression algorithms reduce the size of files without losing any information in the file, which means that we can reconstruct the original data from the compressed file. 

Lossy compression algorithms reduce the size of files by discarding the less important information in a file, which can significantly reduce file size but also affect file quality. 

Zlib - The Node.js Zlib module is used to provide compression and decompression (zip and unzip) functionalities. It is implemented using Gzip and deflate/inflate. 

The zlib module can be accessed using: 

const zlib = require('zlib');   

Compressing and decompressing a file can be done by piping the source stream data into a destination stream through zlib stream. 

To Run the below sample codes for compress and Decompress  

• Please set up a node js project in your system 
• Install node js  
• Install node js zlib module  
• Open node js command prompt window  
• Go to project location where you add zlib_example1.js file  
• Place test.text file on desktop  
• Once you open node js command prompt  
• Run the code using syntax  

Command -   node zlib_example1.js press enter  

Node.js ZLIB Example: Compress File

Let's see a simple example of Node.js ZLIB module to compress a file "test.txt" into "test.txt.gz". 

• const zlib = require('zlib');   
• const gzip = zlib.createGzip();   
• const fs = require('fs');   
• const inp = fs.createReadStream('test.txt');   
• const out = fs.createWriteStream('test.txt.gz');   
• inp.pipe(gzip).pipe(out);  

We have a text file named "test.txt" on the desktop. 

Open Node.js command prompt and run the following code: 

node zlib_example1.js   

C:\Users\FileOpearation\Desktop>node zlib_example1.js 

You can see that it will produce a compressed file named "test.txt.gz" on the desktop. 

Node.js ZLIB Example: Decompress File

Let's see a simple example of Node.js ZLIB module to decompress a file "input.txt.gz" into "input2.txt". 

• const zlib = require('zlib');     
• const unzip = zlib.createUnzip();   
• const fs = require('fs');   
• const inp = fs.createReadStream('input.txt.gz');   
• const out = fs.createWriteStream('input2.txt');   
• inp.pipe(unzip).pipe(out);   

node zlib_example2.js   

Now you will see that same code of "input.txt" is available into "input2.txt" file. 

Note: 

To understand this example well, create "input.txt" file having a large amount of data. Let's assume it has 40 kb data. After compressing this file you will get the size of compressed file "input.txt.gz" to 1 kb only. After decompressing the "input.txt.gz" file, you will get 40 kb of same data into "input2.txt" file. 

Use Cases of Zlib

Compressing HTTP requests and responses

The zlib module can be used to implement support for the gzip and deflate content-encoding mechanisms defined by HTTP. 

The HTTP Accept-Encoding header is used within an http request to identify the compression encodings accepted by the client. The Content-Encoding header is used to identify the compression encodings actually applied to a message. 

Memory Usage Tuning

The memory requirements for deflate are (in bytes): 

The speed of zlib compression is affected most dramatically by the level setting. A higher level will result in better compression, but it will take longer to complete. A lower level will result in less compression, but it will be much faster. 

In general, greater memory usage options will mean that Node.js has to make fewer calls to zlib because it will be able to process more data on each write operation. So, this is another factor that affects the speed, at the cost of memory usage. 

Zlib Properties and Methods

zlib.constants Property – is used to yields an object listing Zlib-related constants. 

Syntax 

Zlib.constants 

Return Value: It returns all the Zlib-related constants. 

zlib.bytesWritten Property – is an application programming interface of zlib module which is used to specify the number of bytes written to the engine before the bytes are processed (compressed or decompressed, as proper for the derived class). 

Syntax 

Zlib.bytesWritten 

Return Value: It returns the number of bytes written to the engine 

zlib.createDeflate() method - is an in-built application programming interface of the Zlib module which is used to create a new Deflate object. 

Syntax: 

Zlib.createDeflate(options) 

Parameters: This method accepts single parameter options which is an optional parameter that holds the zlib options. 

Return Value: It returns a new Deflate object. 

zlib.createDeflateRaw() method - is an inbuilt application programming interface of the Zlib module which is used to create a new DeflateRaw object 

Syntax: 

zlib.createDeflateRaw( options ) 

Parameters: This method accepts single parameter options which is an optional parameter that holds the zlib options. 

Return Value: It returns a new DeflateRaw object. 

zlib.createGunzip() method - is an inbuilt application programming interface of the Zlib module which is used to create a new Gunzip object. 

Syntax: 

zlib.createGunzip( options ) 

Parameters: This method accepts single parameter options which is an optional parameter that holds the zlib options. 

Return Value: It returns a new Gunzip object. 

zlib.gzip() method - is an inbuilt application programming interface of the Zlib module which is used to compresses a chunk of data. 

Syntax: 

zlib.gzip( buffer, options, callback ) 

Parameters: This method accepts three parameters as mentioned above and described below: 

• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• callback: It holds the callback function. 

Return Value: It returns the chunk of data after compression. 

 zlib.createInflateRaw() method - is an inbuilt application programming interface of the Zlib module which is used to create a new InflateRaw object. 

Syntax: 

zlib.createInflateRaw( options ) 

Parameters: This method accepts single parameter options which is an optional parameter that holds the zlib options. 

Return Value: It returns a new InflateRaw object. 

zlib.createInflate() method - is an inbuilt application programming interface of the Zlib module which is used to create a new Inflate object. 

Syntax: 

zlib.createInflate( options ) 

Parameters: This method accepts single parameter options which is an optional parameter that holds the zlib options. 

Return Value: It returns a new Inflate object. 

zlib.createUnzip() method - is an inbuilt application programming interface of the Zlib module which is used to create a new Unzip object. 

Syntax: 

zlib.createUnzip( options ) 

Parameters: This method accepts single parameter options which is an optional parameter that holds the zlib options. 

Return Value: It returns a new Unzip object. 

 zlib.deflate() method - is an inbuilt application programming interface of the Zlib module which is used to compress a chunk of data. 

Syntax: 

zlib.deflate( buffer, options, callback ) 

Parameters: This method accepts three parameters as mentioned above and described below: 

• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• callback: It holds the callback function. 
• Return Value: It returns the chunk of data after compression. 

 zlib.deflateSync() method - is an inbuilt application programming interface of the Zlib module which is used to compresses a chunk of data with Deflate. 

Syntax: 

zlib.deflateSync( buffer, options ) 

Parameters: This method accepts two parameters as mentioned above and described below: 

• buffer: This parameter holds the buffer of type Buffer, TypedArray, DataView, ArrayBuffer, string. 
• options: This parameter holds the zlib options value. 
• Return Value: It returns the chunk of data with deflate. 

zlib.deflateRaw() method - is an inbuilt application programming interface of the Zlib module which is used to compress a chunk of data. 

Syntax: 

zlib.deflateRaw( buffer, options, callback ) 

Parameters: This method accepts three parameters as mentioned above and described below: 

• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• callback: It holds the callback function. 
• Return Value: It returns the chunk of data after compression. 

zlib.deflateRawSync() method - is an inbuilt application programming interface of the Zlib module which is used to compress a chunk of data with DeflateRaw. 

Syntax: 

zlib.deflateRawSync( buffer, options ) 

Parameters: This method accepts two parameters as mentioned above and described below: 

• buffer: This parameter holds the buffer of type Buffer, TypedArray, DataView, ArrayBuffer, string. 
• options: This parameter holds the zlib options value. 
• Return Value: It returns the chunk of data with DeflateRaw. 

 zlib.gunzip() method - is an inbuilt application programming interface of the Zlib module which is used to decompress a chunk of data. 

Syntax: 

zlib.gunzip( buffer, options, callback ) 

Parameters: This method accepts three parameters as mentioned above and described below: 

• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• callback: It holds the callback function. 
• Return Value: It returns the chunk of data after decompression. 

 zlib.gunzipSync() method - is an inbuilt application programming interface of the Zlib module which is used to decompress a chunk of data with Gunzip. 

Syntax: 

zlib.gunzipSync( buffer, options ) 

• Parameters: This method accepts two parameters as mentioned above and described below: 
• buffer: This parameter holds the buffer of type Buffer, TypedArray, DataView, ArrayBuffer, string. 
• options: This parameter holds the zlib options value. 
• Return Value: It returns the chunk of data with Gunzip. 

zlib.unzip() method -  is an inbuilt application programming interface of the Zlib module which is used to decompress a chunk of data. 

Syntax: 

zlib.unzip( buffer, options, callback ) 

• Parameters: This method accepts three parameters as mentioned above and described below: 
• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• callback: It holds the callback function. 
• Return Value: It returns the chunk of data after decompression. 

zlib.unzipSync() method - is an inbuilt application programming interface of the Zlib module which is used to decompress a chunk of data with Unzip. 

Syntax: 

zlib.unzipSync( buffer, options ) 

• Parameters: This method accepts two parameters as mentioned above and described below: 
• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• Return Value: It returns the chunk of data with Unzip. 

 zlib.brotliCompress() method - is an inbuilt application programming interface of the Zlib module which is used to compresses a chunk of data. 

Syntax: 

zlib.brotliCompress( buffer, options, callback ) 

• Parameters: This method accepts three parameters as mentioned above and described below: 
• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• callback: It holds the callback function. 
• Return Value: It returns the chunk of data after compression. 

zlib.brotliCompressSync() method - is an inbuilt application programming interface of Zlib module which is used to compress a chunk of data with BrotliCompress. 

Syntax: 

zlib.brotliCompressSync( buffer, options ) 

• Parameters: This method accepts two parameters as mentioned above and described below: 
• buffer: This parameter holds the buffer of type Buffer, TypedArray, DataView, ArrayBuffer, string. 
• options: It is an optional parameter. 
• Return Value: It returns the chunk of data with BrotliCompress. 

zlib.brotliDecompress() method - is an inbuilt application programming interface of the Zlib module which is used to decompresses a chunk of data with BrotliCompress 

Syntax: 

zlib.brotliDecompress( buffer, options, callback ) 

• Parameters: This method accepts three parameters as mentioned above and described below: 
• buffer: It can be of type Buffer, TypedArray, DataView, ArrayBuffer, and string. 
• options: It is an optional parameter that holds the zlib options. 
• callback: It holds the callback function. 
• Return Value: It returns the chunk of data after decompression. 

zlib.brotliDecompressSync() method - is an inbuilt application programming interface of the Zlib module which is used to decompress a chunk of data with BrotliDecompress. 

Syntax: 

zlib.brotliDecompressSync( buffer, options ) 

• Parameters: This method accepts two parameters as mentioned above and described below: 
• buffer: This parameter holds the buffer of type Buffer, TypedArray, DataView, ArrayBuffer, string. 
• options: This parameter holds the zlib options value. 
• Return Value: It returns the chunk of data with BrotliDecompress. 

Threadpool usage and performance considerations

All zlib APIs, except those that are explicitly synchronous, use the Node.js internal threadpool. This can lead to surprising effects and performance limitations in some applications. 

Creating and using a large number of zlib objects simultaneously can cause significant memory fragmentation. 

Reference –  https://nodejs.org/api/zlib.html#zlib  

In the preceding example, 30,000 deflate instances are created concurrently. Because of how some operating systems handle memory allocation and deallocation, this may lead to to significant memory fragmentation. It is strongly recommended that the results of compression operations be cached to avoid duplication of effort. 

Practice Data Compression  

Data compression is an important aspect of transfer of file which is of very big size with the help of these tools it can reduce the size significantly and transferring of files over networks becomes faster. Practice the different methods explained here and see which one works the best for you. Experience the benefits of data compression for yourself.