How to Use Python cURL?

What is a cURL?

cURL stands for client URL. A cURL is an open-source command-line tool used to create network requests to transfer data across a network. In Python cURL, this data-request transfer to and from servers is done using PycURL. This request tool also helps in testing REST APIs and downloading files. cURL is extremely useful as it is versatile and a low-level command-line tool that offers great performance for transferring data.

Python cURL helps to translate cURL commands into code. The UNIX command cURL helps to send GET, POST, PUT and DELETE requests to a URL. It can be used by importing the requests or pycurl module. The pycurl package is preferred as it supports multiple protocols like FILE, FTPS, HTTPS, IMAP, SMB, SCP, etc. To work with Python cURL you need to install Python, Pip, and the packages Pycurl and certifi. A step-by-step guide for the installations has been provided in the later sections of this article.

Web Scrapping

Web scraping is an automated way to collect a large amount of data from websites. It is often the sole technique to access information on the internet, that is not exposed as downloadables or public APIs. The unstructured data collected from the website is stored in a structured format.

How to Use cURLs with Python?

Having a clear idea about the basic networking concepts such as protocols and client-server communication will help you use cURLs efficiently. The following are required to smoothly work with cURLs.

Python

Download the latest version of Python.
Run the installer after downloading the Python setup.

PIP

Pip is a package management system that works with Python programs. If you are working on an older version of Python there are chances of pip being unavailable.

To install pip:

Open the Command Prompt application.
Enter pip --version.
If pip has been installed, it will show the version number as follows:

Else, the following message will appear:

PycURL

PycURL’s setup.py utilizes curl-config to make sure that the SSL library it’s constructed against is the same one that libcURL. It also ensures that PycURL is running.

To install PycURL type in the following piece of code. pip install pycurl

Certifi

Certifi gives the SSL Mozilla’s root certificates.

In the final step install Certifi using the following piece of code. pip install certifi

How to Make GET Requests?

GET requests are used to fetch pieces of information from HTTP servers. To create a GET request, create a connection between cURL and a web page.

At the initial step, import the modules - BytesIO, pycurl, and certifi.
Declare the objects that are to be used - curlobj, byteobj.
The URL of the desired website is set using the setopt(option, value). method. The 'option' parameter specifies which option to set, e.g. URL, WRITEDATA, etc., and the second parameter specifies the value for that option.
The getvalue() function reads the data retrieved from the BytesIO object in form of bytes and stores it in body.
This body is then decoded and printed in the console.
The output data is displayed in JSON format. In case of errors, the response code is returned.

Code

Output

How to Make POST Requests?

The POST method sends data to the HTTP server and creates or updates data. A POST request (like uploading a file or submitting a form) is sent to the server by enclosing it in the body of the HTTP request.

At the initial step, import the modules - urlencode, pycurl, and certifi.
Declare the object - curlobj.
The data to be posted is stored in dictionary format and the urlencode function encodes this data.
The URL of the desired website is then set using the setopt(option, value). method. The options parameter specifies which option to submit, e.g. POSTFIELDS, and the second parameter specifies the value for that option.

Code

How to Search Responses?

The getinfo() method is used to access more data in an API and search specific responses. Adding the following line of code before curlobj.close() fetches more details from the PycURL docs. print(curlobj.getinfo(curlobj.RESPONSE_CODE)) The above code snippet prints 404 in case of any error and 200 in case of success.

Simple Scrapping

To collect data through web scraping using Python, provide the URL you want to scrape. The web server transmits the data and the executed code helps you to access the HTML or XML page to locate and extract information. This data is then stored in the desired format.

How to Write Response Data to a File?

To write the response data to a file the open() method is used. The open function opens a file in binary mode (i.e. wb) to encode and decode responses easily. The following piece of code will open a file named file.md and add the response to it.

Code

Output

response-data-to-file-output

Examples of cURLs

Python cURL Example 1 - Sending GET Request

Code

Python cURL Example 2 - Sending DELETE Request

Code

Python cURL Example 3 - Writing to a File

Code

FAQs

1. What is cURL?

A: cURL is an open-source command-line tool used to transfer information across networks.

2. What does Pip stand for?

A: Pip stands for Performance Improvement Plan.

3. What is PycURL?

A: PycURL is a Python interface to libcurl used to fetch objects identified by a URL in a Python program.

Learn more

Conclusion

cURL is an open-source command-line tool used to transfer information across networks.
cURL stands for client URL.
The UNIX command curl is used to send PUT, GET, and POST requests to a URL.
Pycurl works faster than the Python library requests that are used for the HTTP requests. Features of Pycurl include multiprotocol support, SSL, authentication, and proxy options.