Node.js Request Module

Introduction

Before diving into the article Let's first understand HTTP, HTTP stands for Hypertext Transfer Protocol. HTTP functions as a request-response protocol in the client-server computing model.

The Node.js request module has been basically designed to make HTTP calls in the simplest way possible. It supports HTTPS and follows redirects by default.

Besides making simple HTTP calls, the Node.js Request module is also used for configuring proxy connections, HTTP streaming, posting data to web forms, providing security to the web pages, basic authentications, etc. We will be looking deeply into each of these uses of the request module and how to perform these operations using the request module in the later part of the article.

Features of Request module

• It is the most widely used and popular module to make HTTP calls.
• It supports HTTPS and follows redirects by default.
• Node.js request module also offers certain convenience methods like request.defaults() and request.post().
• Node.js request module has been designed to be the simplest way possible to make HTTP requests so it can be used in our application very easily.
• Node.js request module can be used to configure a proxy connection.
• Node.js request can also be used to provide security to applications by configuring SSL / TSL Protocol.

Installation of Request Module

You need to install the Node.js request module in your project by NPM (Node Package Manager). You can install the Node.js request module by running the following command on your terminal:

Making HTTP Requests

Well the basic functionality of the request module is to make HTTP requests. So, there are a few ways to make HTTP requests using the Node.js request module. One of the simplest ways to make an HTTP request is shown in the below example:

The above code makes a GET request to the URL: http://localhost:8080/index.html and prints the HTML file on the console. Similar requests can also be made for any other HTML endpoint like: JSON, image(.png, .jpg) etc.

request() mainly has two arguments and the second argument is an optional callback function with three arguments namely error, response and body. For the first argument in request(), it can either be a URL or an options object. Some of the common options widely used in any options object are:

• url: The destination URL of the HTTP request. It is a required option.
• method: The HTTP method to be used (GET, PUT, POST, DELETE, etc)
• headers: It's an object of HTTP headers to be set in the request as key-value pairs.
• form: It's an object containing form data in key-value pairs.

Sample example with an options object as argument in request():

Explanation: In the above example options object has been used to print the JSON file on the console. The options object has a URL, GET method, and headers object. So the JSON object present at the URL is fetched and then JSON.parse() has been used to convert the JSON object to a javascript object to make it print on the console.

The request format in the above example can be used for any type of HTTP method, whether it's GET, PUT, POST, DELETE, etc. But in some cases like in PUT and POST methods, we can include the data within the request. Some of the ways by which the data can be sent are:

• body: It's a Buffer, String, or Stream object (can be an object if JSON option is set to true)
• form: It's an object of simple key-value pair data used in the case of simple URL Encoded forms.
• multipart: An array of objects that can contain their own headers and body attributes through this different file can also be uploaded.

Forms

You must have come across several web forms or several websites asking you for your response as certain form and submit them. Well to submit these response data through your bot or while interfacing with REST APIs in your application, you need Node.js request module in your project. Node.js request module has certain ways to help us with these taks.

For simple regular forms i.e., URL-encoded (application/x-www-form-urlencoded) forms, You can simply use .post() in your program for form uploads.

Example for application/x-www-form-urlencoded (URL-Encoded) form uploads:

The above code will upload data in application/x-www-form-urlencoded (URL-Encoded Forms) forms just like an HTML form would. But in URL-Encoded Forms you can't upload files. In order to do that, you need to have multipart/form-data (Multipart Form Uploads) forms.

For multipart/form-data forms, we use the form-data library. By the formData option you can pass your upload. Using formData option you can now pass file data to the server via Buffers, Streams, or even non-file data (as before) with simple key-value pairs.

Example for multipart/form-data form uploads:

will send your files with a MIME type of multipart/form-data, which is a multipart form upload.

Streams

HTTP Streaming is a push-style data transfer technique that allows a web server to continuously send data to a client over a single HTTP connection that remains open indefinitely. In simple words using a stream, we can send the file contents to the user as it's being loaded from disk, byte by byte. Since we don't have to wait for the entire file to load, this reduces the time taken by the server to respond to the user's request. Streaming also reduces the memory usage required to handle the users' requests since we don't need to load the entire file at once.

You can stream any response to a file stream as:

Example:

You can also stream a file to a PUT or POST request. This method will also check the file extension against a mapping of file extensions to content-types (in this case application/json) and use the proper content-type in the PUT request (if the headers don’t already provide one).

Example:

Request can also pipe to itself. When doing so, content-type and content-length are preserved in the PUT headers.

Example:

Proxies

An HTTP Proxy serves two intermediary roles as an HTTP Client and an HTTP Server for security, management, and caching functionality. Then, any traffic that is processed through the server will appear as though it came from the proxy’s dedicated IP address instead of the one that your device is associated with. Additionaly we can also manage web traffic, web traffic using HTTP Proxies. So to include Proxies in your program, the simplest way to achieve this is by using the proxy option, which takes an address in which the traffic is proxied through:

Example:

The options object is one way to specify a proxy, but request also uses the following environment variables to configure a proxy connection:

• HTTP_PROXY / http_proxy
• HTTPS_PROXY / https_proxy
• NO_PROXY / no_proxy

When HTTP_PROXY / http_proxy are set, they will be used to proxy non-SSL requests that do not have an explicit proxy configuration option present. Similarly, HTTPS_PROXY / https_proxy will be used for SSL requests that do not have an explicit proxy configuration option. And NO_PROXY / no_proxy variable is used to specify the sites which should not be proxied.

Fuel Your Full-Stack Passion! Enroll in Our Full Stack Developer Course Taught by Industry Experts and Get Certified!

TLS / SSL

SSL (Secure Sockets Layer) is the standard technology for keeping an internet connection secure and safeguarding any sensitive data that is being sent between two systems, preventing criminals from reading and modifying any information transferred, including potential personal details. TLS (Transport Layer Security) is just an updated, more secure, version of SSL. HTTPS (Hyper Text Transfer Protocol Secure) appears in the URL when a website is secured by an SSL certificate.

We need to look into two possible scenarios where we would require TLS / SSL Protocol to be used in our project:

1. Sometimes an API needs to have some extra security and therefore requires a client certificate. This is actually fairly common with private corporate APIs.
2. You want your HTTP requests to explicitly trust certain certificate authorities, which could include certificates self-signed by you or your company.

So to include TLS / SSL Protocol in our project we can set it in the options object similar to all of the other configurations we've seen so far i.e., we can directly set TLS / SSL Protocol options, such as cert, key and passphrase in the options object.

Example:

Redirects

You may have come across various cases where redirection is being used. You can understand redirection as a technique to move the visitors to a different Web page than the one they request. In some applications like web scraping (where bots to extract content and data from a website), you need to follow redirects in order for your request to be successful.

Using Node.js request module we can follow redirects and it is quite easy to observe and trace all the intermediate urls before the final url is reached. We can do that using followRedirect option, which takes either true or false, or a function that should return a boolean value. The function takes a single argument response, that we can use to log the url the request is currently visiting. There is an option to specify whether to follow redirects by default, but request module goes even one step further and will let you include a function that can be used to conditionally determine if the redirect should be followed or not.

Example of redirects in request module of Nodejs:

Output:

The redirect options available to us are:

• followRedirect - follow HTTP 3xx responses as redirects (default: true). This property can also be implemented as function which gets response object as a single argument and should return true if redirects should continue or false otherwise.
• followAllRedirects - follow non-GET HTTP 3xx responses as redirects (default: false)
• followOriginalHttpMethod - by default we redirect to HTTP method GET. you can enable this property to redirect to the original HTTP method (default: false)
• maxRedirects - the maximum number of redirects to follow (default: 10)
• removeRefererHeader - removes the referer header when a redirect happens (default: false).

Basic Authentications

You may come across sites with basic authentication i.e., credentials include only username and password. Such sites which uses basic access authentication can still be accessed using the auth option in option object.

Example:

If passed as an option, auth should be a hash containing values:

• user / username
• pass / password
• sendImmediately (optional)
• bearer (optional)

You can also achieve basic authentication to the sites using the auth method which takes parameters as auth(username, password, sendImmediately, bearer).

Example:

Convenience Methods

For our ease and comfort, there are some shorthand methods for different HTTP METHODs and some other conveniences. So let's look into some of those:

request.defaults(options): This method returns a wrapper around the normal request API that defaults to whatever options you pass to it.

Debug the Operation of Request

Error handling or debugging is an essential part of making requests to external APIs, as we can never be sure that what will happen to those requests. Apart from our client errors the server may respond with an error or just send the data in a wrong or in an inconsistent format. We must keep these in mind when we try handling the response.

There are at least three ways to debug the operations of a request:

• Launch the node process like NODE_DEBUG=request node script.js (lib, request, otherlib works too).
• Set require('request').debug = true at any time (this does the same thing as #1).
• Use the request-debug module to view request and response headers and bodies.

Request Has Been Deprecated

The Node.js request module package has been deprecated. It means that new releases of Node.js request module will not come with new features.

The reason behind such step is that since request was one of the first modules added to the npm registry. So as npm grew so did dependence on request. Now several modules in NPM have certain dependency on request module so the developers of this package decided to take this step as it’s very hard to change request package in any meaningful way as the change may not be adopted by the majority of its dependents.

Take your first step towards becoming a Node.js pro. Enroll in our Free Node JS certification course & bring your JavaScript skills to a whole new level.