PyTorch Tutorial

PyTorch is an open-source deep-learning library based on Torch, a framework for scientific computing.

With most of it written in C++ and CUDA, PyTorch offers high-performance and efficient scientific computing through its core data structure called tensors. It is flexible in its use, highly customizable, and thus allows for the implementation of novel neural architectures with ease.

This PyTorch tutorial series takes a step-by-step approach to learning PyTorch, from the basic question "What is PyTorch" to building advanced neural nets in PyTorch.

Audience

Since PyTorch is one of the most popular and easy-to-use deep learning libraries both in research and academia, and industry, this curriculum is targeted at deep learning enthusiasts who want to leverage machine learning to build powerful AI-powered applications and/or researchers who want to research and design new techniques to build novel and state of the art architectures for deep learning applications.

In this PyTorch tutorial series, we detail the main features of PyTorch necessary to learn and elaborate on its various aspects, describing how powerful the library is.

Prerequisites

To be able to learn and follow the PyTorch tutorial series, one needs to have the following as prerequisite knowledge:

• Basic Python:
  
  We will be learning to use the library from its Python interface, so a basic knowledge of how to program in Python is required to learn PyTorch. Just some beginner to intermediate level of proficiency is enough as skills can be picked up on the go. The official Python documentation is a great place to refer to.
• Basic knowledge of Deep Learning:
  
  While one does not need to be an expert in deep learning theory and the mathematics behind it, a basic understanding of what deep learning deals with as a field is good to have to learn PyTorch using this PyTorch tutorial series. We will be covering the basic concepts hands-on in the curriculum, nevertheless, so feel free to start and learn on the go.

PyTorch Features

Let us talk about the main features of PyTorch to understand the fundamental question, "What is PyTorch?"

These features make PyTorch one of the top choices by deep learning practitioners in research and industry alike.

• Easy Interface:
  The library has an easy-to-use interface in Python which is the most popular language for building machine learning languages.
• Python usage:
  PyTorch's use in Python is super easy as it is very Pythonic in how the API can be accessed from Python.
• Computational graphs:
  The automatic differentiation engine of PyTorch called autograd deals with building dynamic computational graphs during the run time, which saves the user from pre-specifying the graph structure, unlike frameworks like tensorflow where the graphs are static in nature. This is particularly useful when control flow is used in the forward pass of a neural network computation.

The Levels of Abstraction in PyTorch

Tensor

Tensor is the primary data structure in PyTorch. It is the data structure in which every data in pytorch is stored and represented, and so is what's expected by almost every computation using the components of the PyTorch library, be it the loss computation, the forward pass, the gradient updation, etc.

Modules

PyTorch offers access to several commonly used layers in building deep neural networks like batchnorm, self-attention module, recurrent neural layers, convolutional layers, LSTMs, etc., all of which inherit from the base class nn.Module.

Access to these lower-level layers is the key to being able to implement and experiment with novel architectures in PyTorch.

Advantages of PyTorch

The library has several advantages over other existing deep-learning libraries.

• The foremost advantage is its ease of use which is appealing to beginners and experts in the field alike.
• It allows access to lower-level building blocks of neural networks while being as abstract as possible, which makes it a popular choice among deep learning researchers as it facilitates the easy building of novel architectures.
• The Pythonic API makes it easy for a broader category of the audience to grasp the library in a very small amount of time.
• The library is continuously updating and adding new features for fast and efficient training of neural networks (the latest PyTorch 2.0 is an example). With massive support means of the official PyTorch developer forums, a lot of help is available for debugging the code written in PyTorch.

TensorFlow vs. PyTorch

Let us look at some key pointers to highlight the differences between the two libraries:

• Parallelism:
  While both TensorFlow and PyTorch offer support for distributed training over multiple GPUs, PyTorch allows it more easily.

In a single line of code, one could wrap the model inside PyTorch's DataParallel class to enable Parallelism, while it takes writing more code to avail the same in TensorFlow.

• Deployment:
  No matter how shiny or accurate, a deep learning model shall be useless just sitting in a jupyter notebook. In fact, it's a waste of time and money resources if models aren't brought to life, i.e., production.

When it comes to deployment, TensorFlow has been offering easy and efficient tools (TensorFlow Serving and TensorFlow Lite) for the deployment of models on cloud servers and mobile devices.

While PyTorch initially lacked in this arena, it has now worked remarkably toward closing this gap by the introduction of deployment tools like TorchServe, which was introduced in 2020 and supported both REST and gRPC APIs, and PyTorch Live, which supports the building of iOS and Android AI-powered apps. PyTorch Live also plans to support audio and video inputs.

• Ease of use:
  Keras, an open-source library providing a Python API for training neural networks, was released in 2015 and became popular because of its easy-to-use nature.

But, Keras didn't offer much access to low-level features and thus lacked the customizability that researchers needed to implement novel architectures. TensorFlow was released in late 2015 with the goal of solving this problem.

Still, TensorFlow's non-Pythonic and somewhat difficult-to-implement nature was solved by the release of PyTorch. PyTorch, along with offering a Pythonic style of coding, gives easy access to low-level features allowing for greater customizability that facilitates ease of implementation of novel architectures.

Why should you Learn PyTorch?

Artificial intelligence, specifically deep learning, has seen massive success in a wide variety of domains ranging from facial detection systems to speech and language processing. Research and development in this area are growing at an exponentially massive scale and will continue to do so in the coming years.

Since PyTorch is one of the most popular and, more importantly, easy-to-use deep learning libraries, learning PyTorch is very beneficial as it empowers developers to build AI-powered applications and scale them to serve the users.

How Long does it Take to Learn PyTorch?

For someone with a basic to intermediate level of experience in programming in ideally any language but preferably Python, it is very easy to learn PyTorch within a span of 2-3 weeks with a hands-on approach focussed on building deep learning systems in PyTorch.

Is it Hard to Learn PyTorch?

PyTorch has a very Pythonic nature, and it smoothly integrates with Python, which is the most widely used language for data science. PyTorch requires only a basic understanding of the principles of Object Oriented Programming (OOPs) and does not require one to be an expert at it.

Most skills can be picked on the way as one builds and debugs neural nets in it.

Along with it, there is massive community support by means of the official PyTorch documentation, Github issues, and the very helpful PyTorch developer forums. All of this makes it very easy to learn PyTorch and debug it along the way.

How to Learn PyTorch: Step-by-Step

• The Scaler PyTorch curriculum provides a step-by-step guide to developers or students to learn PyTorch in an easy manner.
• We present sufficient theoretical details along with practical hands-on code examples in every article within the curriculum.
• A lot of topics are included in the curriculum, from basic things like discussing "What is PyTorch" and installation of PyTorch to building actual models in it and everything in the between.
• We recommend going through every article of this PyTorch tutorial series in order while coding and following all the code examples all along and experimenting with them to be able to make the most out of this curriculum and learn PyTorch in an easy and fun manner.

About this PyTorch Curriculum

The PyTorch curriculum by Scaler is a step-by-step resource that assumes no knowledge of PyTorch and teaches various components of the PyTorch ecosystem in an easy-to-follow and structured manner.

It starts by explaining the basic tensor data structure and teaches how to store and manipulate data using it. After that, we move on to learn the basic components of a machine learning cycle all hands-on in PyTorch - the data (accessing and loading), the model definition and model building, model training, and so on and so forth.

Higher-level concepts like parallel training of neural networks across multiple GPUs, memory management, and profiling are also included. All of the tutorials within the curriculum are hands-on, with appropriate projects and code samples to teach the end-to-end cycle of building particular types of neural networks in PyTorch.

Apart from this, the official PyTorch documentation should be referred to learn about the library holistically.

Take-Away Skills from this PyTorch Tutorial

After following along with the PyTorch curriculum thoroughly, the user should be comfortable enough with the library to build novel architectures in it with ease, train them and use the trained networks for inference. The curriculum is enough to make the user comfortable with what the PyTorch ecosystem has to offer and teaches adequate skills to follow along with the official documentation to build upon them and debug the code without much hassle.