Python Download File From URL

Python Download File from URL

There are many methods to download a file from a URL in python. In this session, we will see a few of them.

Requests Library

The first method in python download file from URL is to use Requests, which is an HTTP library in python with various applications and one of its applications is to download a file from a URL. We will use the requests.get() method of the requests library. First of all, you will need to install the requests library using the command:

Or you can download it manually from their official website. Now, you can import it to your project using import requests and use the requests.get() method to download the data behind that URL by passing it as a parameter to the get() method. The response generated can be written to a file in your system by calling the open(). Let's see the python program to do this.

Here, 'wb' means, that you are opening the file for writing purposes in a binary format and the output of this code will be an image in the same folder where your python program file is. For this method, all we need is the URL of the image source which we want to download.

wget Module

Another method to do this is to use the wget module. We will first need to install it using the command:

Post this, you can import the wget module into your project and can use wget.download() method of the wget module to download a file from a specific URL and save it on your local machine. The python program to do this is:

Here, we have passed the specified URL and the image file name locally where we want to download the image as the parameters to this wget.download() method. This will download a file in the image.jpg file.

urllib Module

Another common method to do this is to use the urlib module. To install it, you can use the command:

Now, you can import the urllib module into your project using import urllib. After importing the module, you can use the urllib‘s request.urlretrieve() method to download a file from a specific URL and save it on your local machine. Let's see the python program to do this operation:

As specified above, the request.urlretrieve() method is to download a file from a specific URL. So, these were the three most common methods to download a file from a URL in python. Now, in the upcoming section, we will see how we can download large files using the REQUESTS method.

Downloading Large Files

As we have seen to download a file from a URL using the requests library in python we can use the piece of code:

Here, response.content is a string which is basically storing the file data for a while and then opens it in the specified file using open(). But let's say the data consists of large files, then, it won’t be possible to save all the data in a single string, and will cause the overflow. To overcome this problem, we will do some slight changes to our previous program:

Since all file data can’t be stored by a single string, we will use the response.iter_content method to load the complete data in chunks (a piece of something), by specifying the chunk size which is passed as a parameter. Also, we have to set the stream parameter to True as this will cause the download of response headers only and the connection will remain open until the response.iter_content method is loading. This will help in avoiding the reading of the content all at once into memory for large responses. A fixed chunk (according to the chunk size provided) will be loaded each time while response.iter_content is iterated.

Let's see the example for this:

This will help in reducing the overflow on the response.content string.

Downloading Videos in Python

To download a list of videos we first need to extract all the video links and then we can download each of the videos using the requests library. If we have to download a single video then we can directly use the URL of that video to download it.

The above program will download the video one by one and write them over a file, using the requests library. As video is mostly considered as a large file data, so we have divided it into smaller chunks using the method discussed in the above section.

Advantages of Using REQUESTS library in Python for Downloading Files

• The REQUESTS library in Python helps in easily downloading large files, and videos as we saw above.
• It abstracts or hides the complexities of making requests behind a simple API so that you can focus on your task and consume data in your application.
• This is a browser-independent method that makes it accessible.
• It is a much faster library.
• Requests library can be used to scrape the data from the website and hence, can download all files by using the above code. This will help us to avoid the tiring manual task of downloading videos one by one.
• The requests library can be used to download the entire directories by recursively going through the website.
• This library provides us with a very simple API for interacting with HTTP operations such as GET, POST, etc.
• The handling of cookies and sessions in this library is very easy.
• The security is also taken care of in this library with the help of authentication module support.