Buffers and String Decoders in Node.Js

Introduction to Buffers in Node.js

A buffer is a memory space (usually in RAM) with binary data. The built-in Buffer class in Node.js allows us to access these memory spaces. Buffers, like arrays in JavaScript, store a sequence of integers. A buffer, unlike an array, cannot be resized once it is created. A buffer is a memory section used to temporarily store data. A buffer is generally used between devices of different speeds so that they can continue to operate at their respective speeds without data loss.

A binary stream is a large collection of binary data. Binary streams are not sent together due to their massive size. Instead, they are cut into smaller pieces before being sent. Excess data is stored in a buffer until the data processing unit can accept more data streams. Node.js includes the Buffer class, which stores raw data like an array of integers but corresponds to a raw memory allocation outside the V8 heap. Because pure JavaScript is not compatible with binary data, the Buffer class is used. So, while dealing with TCP streams or the file system, octet streams must be handled.

Node.js servers frequently need to do filesystem operations. The files are stored in binary. Node.js also handles TCP streams. They are used to provide secure connections to receivers before sending binary data in small chunks. Streams of data provided to the receiver must be saved until the receiver accepts more chunks of data for processing. The Node.js buffer class comes into play here. Outside of the V8 engine, it handles and stores binary data.

If you've already written Node.js code, you may have utilized buffers indirectly. When you use fs.readFile() to read from a file, the data provided to the callback or Promise is a buffer object. Furthermore, Node.js HTTP queries return data streams that are temporarily held in an internal buffer when the client cannot digest the stream all at once.

Node.js Creating Buffers

A Node buffer can be built in a variety of ways. The three most common ways are as follows :

• Create an uninitiated buffer: Following is the syntax of creating a buffer of 15 octets without any initialization.

• Create a buffer from the array: The syntax for creating a Buffer from an array is as follows.

• Create a buffer from string: The syntax for creating a Buffer from a given string and optionally encoding type is as follows.

Different Types of Node.Js Buffer Module Methods

• Buffer.alloc(): A buffer object of a specified size or length can be created with the help of this method. The value given as fill will be initialized. The Buffer.alloc() method has following syntax :

  Description of the parameters :

  • size: It is an integer that gives the length desired for the new Buffer.
  • fill: The value that is used to prefill the buffer. The default value is 0. It accepts any of the following: integer, string, buffer type of data.
  • Encoding: It is a string. If fill is a string, this is its encoding. Default: 'utf8'.

  For example,

  The above code will give the following output :

• Buffer.allocUnsafe(): It creates a buffer without initializing the values. As a result, the contents of the newly formed buffer are unknown, posing a security risk because it may include sensitive or secret data.

  The Buffer.allocUnsafe() method has following syntax :

  Description of the parameters :

  • size: It is an integer that gives the length desired for the new Buffer.

  For example,

  The above code will give the following output :

• Buffer.from(): This method is used to convert a string, object, array, or buffer. The Buffer.from() method has following syntax :

  Description of the parameters :

  • string: The data that is to be converted.
  • encoding: The encoding of string data. Default is utf8. For example,

  The above code will give the following output :

• buf.compare(): This method can be used to compare two buffers. It gives:

  • 0 : If both buffers are the same
  • 1 : If the target buffer comes before the source buffer.
  • -1 : If the source buffer comes before the target buffer.

  The buf.compare() method has following syntax :

  Description of the parameters :

  • target : target buffer.
  • targetStart: Position from where the comparison begins on the target buffer.
  • targetEnd: Position on which comparison ends on the target buffer.
  • sourceStart: Position from where the comparison begins on the source buffer.
  • sourceEnd: Position on which comparison ends on source buffer.

  For example,

  The above code will give the following output :

• Buffer.concat(): This method is used to merge two or more buffers. The Buffer.concat() method has following syntax :

  Description of the parameters :

  • list: List of buffers that are to be concatenated.

  For example,

  The above code will give the following output :

• buf.copy(): This method copies a given number of bytes from the source buffer to the target buffer.

  The buf.copy() method has following syntax :

  Description of the parameters :

  • get : buffer to which we need to copy.
  • targetStart : Position from where copy starts from source.
  • sourceStart : position from which copy starts.
  • sourceEnd : Position till which copy is done.

  For example,

  The above code will give the following output :

• buf.equals(): This method is used to perform equality checks on two buffers. If the buffers match then it returns true, otherwise, it returns false. The buf.equals() method has following syntax:

  Description of the parameters :

  • secondBuffer : Buffer to compare with.

  For example,

  The above code will give the following output :

• buf.fill(): This method is used to initialize the buffer with the value given. The buf.fill() method has following syntax :

  Description of the parameters:

  • value : The value with which the buffer will be filled.

  For example,

  The above code will give the following output:

• buf.indexOf(): This method determines whether the buffer has a given value. If the value is present, it will return the index of the value's first occurrence, otherwise, it will return -1.

  The buf.indexOf() method has following syntax:

  Description of the parameters :

  • value : the value whose index is to be found.

  For example,

  The above code will give the following output :

• buf.length: This property is used to return the length of the buffer object.

  The buf.length property has the following syntax :

  For example,

  The above code will give the following output:

• Buffer.slice() : This method is used to create a new buffer using the slice of the old buffer from the start and end points supplied.

  The buf.slice() method has following syntax:

  Description of parameters :

  • tart: starting offset of the new buffer.
  • end: ending offset of the new buffer.

  For example,

The above code will give the following output :

• buf.toJSON(): This method is used in the conversion of the buffer into JSON.

  The buf.toJSON() method has following syntax:

  For example,

  The above code will give the following output :

• buf.toString(): This method is used in the conversion of the buffer into the string.

  The buf.toString() method has following syntax:

  Description of the parameters:

  • encoding: Character encoding used for decoding.
  • start : starting offset.
  • end : ending offset.

  For example,

  The above code will give the following output :

What Is String Decoder Module in Node.Js

To convert a buffer into a string, the Node.js StringDecoder is used. It is similar to buffer.toString() but adds UTF support. The node:string decoder module implements an API for decoding Buffer objects into strings while preserving encoded multi-byte UTF-8 and UTF-16 characters.

StringDecoder is a built-in Node.js module, so we don't need to install it. You can simply import it with the following statement.

or

Usages of String Decoders

The buffer data is decoded into a string using the NodeJS StringDecoder. Buffers are used to store streams of binary data. Because binary data can only be understood by computers, it must be converted into text so that users can understand it.

The buffer class has a method buffer.toString(), which is also used to transform buffer data into a string. However, StringDecoder provides additional support for UTF encoding.

Different Types of Methods in Strings Decoders

StringDecoder provides two methods to decode a buffer into a string, the write() method and the end() method, each with slightly different capabilities.

1. write(): This method decodes the data supplied as an argument from the buffer and returns the decoded text. This function skips the incompleted byte characters at the end of the buffer stream and stores them in the internal buffer for the next StringDecoder method call.

   Syntax:

   Where the buffer can be a standard JavaScript buffer, TypedArray, or Data that contains the bytes that have to be decoded.

   Return Value : It returns a decoded string from the provided buffer.

   Example: Let us begin by importing the StringDecoder module, then build an object by supplying the utf-8 encoding to the StringDecoder's constructor’s parameter. The write() method is invoked using this object.

   Using the Buffer.from() method, you can create a new buffer.

   We pass a string as an input to this function, which uses that string to initialize the new buffer. Then, pass the buffer as a parameter to the write() method and set it to a variable.

   In the end, print the result.

   Here is the combined code for the write() function :

   Output:

2. end(): The stringDecoder.end() method is used to return as a string all of the remaining input stored in the internal buffer. This method guarantees that any missing UTF-8 and UTF-16 characters are replaced with appropriate substitution characters for the character encoding.

   When a buffer argument is supplied, the stringDecoder.write() method is run once with the data before the remaining buffer input is returned.

   Syntax:

   Buffer: The bytes that must be decoded are contained in a Buffer, TypedArray, or DataView. It is an optional parameter.

   Return Value: It returns the string representation of the remaining input stored in a buffer.

   Example:

   Let us begin by importing the StringDecoder module, then build an object by supplying the utf-8 encoding to the StringDecoder's constructor’s parameter. The end() method is invoked using this object.

   Then in the write() method of StringDecoder pass a buffer that makes use of the [0xE2, 0x82, 0xAC] as an argument that represents the euro symbol.

   Then, using the end() method, you can retrieve the data contained in the internal buffer by decoding that data into a string that represents the euro symbol.

   Here is the combined code for the end() function :

   Output: