You have decided to learn deep learning and have learned some theory and want to do some hands-on practice now, but you are in a dilemma of which framework to start with (Tensorflow or PyTorch)? Don’t worry, I have been there too and that’s the reason I am starting with this sequence where I will be writing a series of articles that can help you get started with PyTorch!
So, let’s get started!
The flow of the article:
- A brief introduction to PyTorch
- Tensors
- Initializing a tensor
- Attributes of tensors
- Operations on tensors
- PyTorch-NumPy interoperability
- GPU/CPU compatibility
What is PyTorch?
PyTorch is an open-source machine learning framework based on the Torch library, used for applications such as computer vision and natural language processing, primarily developed by Facebook’s (now Meta) AI Research lab. It allows maximum flexibility and speed in scientific computing for deep learning. Also, it is a replacement for NumPy to use the power of GPUs.
Why PyTorch?
According to my research, TensorFlow, Keras, and PyTorch are the most popular libraries mentioned in the machine learning community. Tensorflow suits well if you want to develop models for production, develop models which need to be deployed on mobile platforms, and need to use large-scale distributed model training. Whereas PyTorch works well for research purposes and if you want everything Pythonic (like me😉)!
Tensors
Tensors are specialized data structures that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters. Tensors are similar to NumPy’s ndarrays, the only difference is they can run on GPUs and other hardware accelerators. For example, 1d-tensor is a vector, 2d-tensor is a matrix, 3d-tensor is a cube, and 4d-tensor is a vector of cubes. If you want to understand it visually, here is an amazing Youtube video in which Daniel Fleisch, a professor in the Department of physics at Wittenberg University has explained it fantastically.
Initializing a tensor:
Let’s take a look at how to create a tensor in PyTorch. Tensors can be initialized in various ways. Let’s import the necessary libraries:
import torch
import numpy as np
1. Directly from data:
data = [[1, 2],[3, 4]]
tensor_data = torch.tensor(data)
# Example:
print(tensor_data)
# Output:
tensor([[1, 2],
[3, 4]])
2. Create a tensor with random values:
random_tensor = torch.Tensor(n,m) # where n is the number of rows and m is the # number of columns (n x m)
# Example:
random_tensor = torch.Tensor(2,3)
print(random_tensor)
# Output:
tensor([[6.8664e-44, 7.7071e-44, 1.1771e-43],
[6.7262e-44, 7.9874e-44, 8.1275e-44]])
3. Create a tensor with random values between a and b:
uniform_tensor = torch.Tensor(n, m).uniform_(a, b) # where a and b are any natural numbers
# Example:
uniform_tensor = torch.Tensor(3,3).uniform_(1,5)
print(uniform_tensor)
# Output:
tensor([[2.2433, 3.1906, 4.2577],
[3.6101, 1.9020, 2.6306],
[1.2986, 3.6643, 1.4555]])
4. Create a tensor filled with random numbers from a uniform distribution on the interval [0, 1):
rand_tensor = torch.rand(n, m)
# Example:
rand_tensor = torch.rand(3, 3)
print(rand_tensor)
# Output:
tensor([[0.6445, 0.5295, 0.8926],
[0.6845, 0.2812, 0.7323],
[0.9651, 0.4299, 0.8283]])
5. Create a zeros tensor of size n x m:
zero_tensor = torch.zeros(n, m)
# Example:
zero_tensor = torch.zeros(3, 3)
print(zero_tensor)
# Output:
tensor([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])
6. Create an ones tensor of size n x m: ones_tensor = torch.ones(n, m)
# Example:
ones_tensor = torch.ones(3, 3)
print(ones_tensor)
# Output:
tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]])
Attributes of a tensor:
Let’s see how we can get the attributes of a tensor like shape, data type, and on which device they are stored.
tensor = torch.randn(n, m)
print("Shape of a tensor: {}".format(tensor.shape))
print("Data type of a tensor: {}".format(tensor.dtype))
print("Dimension of a tensor: {}".format(tensor.dim()))
print("Device tensor is stored on: {}".format(tensor.device))
# Example:
tensor = torch.randn(3, 3)
print("Shape of a tensor: {}".format(tensor.shape))
print("Data type of a tensor: {}".format(tensor.dtype))
print("Dimension of a tensor: {}".format(tensor.dim()))
print("Device tensor is stored on: {}".format(tensor.device))
# Output:
Shape of a tensor: torch.Size([3, 3])
Dimension of a tensor: 2
Data type of a tensor: torch.float32
Device tensor is stored on: cpu
Operations on Tensors:
There are many operations on tensors, out of which we will discuss the most common and important ones over here. If you are interested to know about all operations — they are discussed here.
1. Indexing operation: Using this operation, you can access or replace elements in a tensor.
tensor = torch.Tensor([[1, 2], [3, 4]])
# Replace an element at position 0,1
tensor[0][1] = 7
print(tensor)
# Output:
tensor([[1., 7.],
[3., 4.]])
# Access an element at position 1,0
print(tensor[1][0])
print(tensor[1][0].item()) # using .item() you can access the scalar object
# Output:
tensor(3.)
3.0
2. Slicing operation: Using this operation, you can access elements in a specific index range.
tensor = torch.Tensor([[1, 2], [3, 4], [5,6]])
# First element of every row
print(tensor[:,0])
# Output:
tensor([1., 3., 5.])
# Last element from every row
print(tensor[:,-1])
# Output:
tensor([2., 4., 6.])
# All elements from second row
print(tensor[1,:])
# Output:
tensor([3., 4.])
# All elements from last two rows
print(tensor[1:,:])
# Output:
tensor([[3., 4.],
[5., 6.]])
3. Joining or concatenating a tensor: This operation is used to concatenate a sequence of tensors along a given dimension.
tensor_1 = torch.Tensor([[1, 2], [3, 4], [5,6]])
tensor_2 = torch.Tensor([[8,9],[10,11],[12,13]])
new_tensor = torch.cat([tensor, ten2], dim=1)
print(new_tensor)
# Output:
tensor([[ 1., 2., 8., 9.],
[ 3., 4., 10., 11.],
[ 5., 6., 12., 13.]])
There is one more operation similar to joining, known as torch.stack.
4. Reshaping a tensor: Using this operation, you can reshape your tensor to n x m size.
tensor = torch.Tensor([[1, 2], [3, 4]])
reshaped_tensor_1 = reshape_tensor.view(1,4)
reshaped_tensor_2 = reshape_tensor.view(4,1)
print("Reshaped tensor 1: ",reshaped_tensor_1)
print("\nReshaped tensor 2: ",reshaped_tensor_2)
# Output:
Reshaped tensor 1 tensor:
([[1., 2., 3., 4.]])
Reshaped tensor 2 tensor:
([[1.],
[2.],
[3.],
[4.]])
5. Transpose: You can perform a transpose operation on a tensor by using .t() or .permute[-1, 0).
tensor = torch.Tensor([[1, 2], [3, 4], [5,6]])
print("Tensor: ",tensor)
print("\nTensor after transpose: ",tensor.t())
# Output:
Tensor: tensor([[1., 2.],
[3., 4.],
[5., 6.]])
Tensor after transpose: tensor([[1., 3., 5.],
[2., 4., 6.]])
Transpose using permute function:
tensor = torch.Tensor([[1, 2], [3, 4], [5,6]])
print("Tensor: ",tensor)
print("\nTensor after transpose: ",tensor.permute(-1,0))
# Output:
Tensor:
tensor([[1., 2.],
[3., 4.],
[5., 6.]])
Tensor after transpose:
tensor([[1., 3., 5.],
[2., 4., 6.]])
6. Cross-product: You can get the cross product of two tensors by using either of the two:
tensor_1.cross(tensor_2) or torch.cross(tensor_1, tensor_2)
tensor_1 = torch.Tensor([[1, 2], [3, 4], [5,6]])
tensor_2 = torch.Tensor([[8, 9],[10, 11],[12, 13]])
cross_prod = tensor_1.cross(tensor_2)
print(cross_prod)
# Output:
tensor([[-14., -14.],
[ 28., 28.],
[-14., -14.]])
7. Matrix product: Performs a matrix multiplication of the two tensors. If tensor_1 is an (n×m) tensor, tensor_2 is an (m×p) tensor, then the output will be (n×p) tensor.
tensor_1 = torch.Tensor([[1, 2, 3], [3, 5, 6], [7, 8, 9]])
tensor_2 = torch.Tensor([[10, 11, 12],[13, 14, 15],[16, 17, 18]])
# Using tensor_1.mm(tensor_2)
matrix_prod = tensor_1.mm(tensor_2)
print(matrix_prod)
# Output:
tensor([[ 84., 90., 96.],
[191., 205., 219.],
[318., 342., 366.]])
# Using torch.mm(tensor_1, tensor_2)
maxtrix_prod = torch.mm(tensor_1,tensor_2)
print(matrix_prod)
# Output:
tensor([[ 84., 90., 96.],
[191., 205., 219.],
[318., 342., 366.]])
8. Element-wise multiplication: This operation performs the element-wise product of the two tensors.
tensor_1 = torch.Tensor([[1, 2, 3], [3, 5, 6], [7, 8, 9]])
tensor_2 = torch.Tensor([[10, 11, 12],[13, 14, 15],[16, 17, 18]])
product = tensor_1.mul(tensor_2)
print(product)
# Output:
tensor([[ 10., 22., 36.],
[ 39., 70., 90.],
[112., 136., 162.]])
9. Sum of a tensor: This operation returns the sum of all the values of a tensor.
tensor_1 = torch.Tensor([[10, 11, 12],[13, 14, 15],[16, 17, 18]])
agg = tensor_1.sum()
agg_item = agg.item()
print(agg_item)
# Output:
126.0
10. In-place addition operation: This operation adds a scalar value to each and every element of the tensor.
tensor_1 = torch.Tensor([[1, 2, 3], [3, 5, 6], [7, 8, 9]])
print(tensor_1)
tensor_1.add_(5)
print("\n",tensor_1)
# Output:
tensor([[1., 2., 3.],
[3., 5., 6.],
[7., 8., 9.]])
tensor([[4., 5., 6.],
[ 6., 8., 9.],
[10., 11., 12.]])
PyTorch-NumPy interoperability:
You can easily convert a NumPy array to a PyTorch tensor and vice-versa.
1. PyTorch tensor to a NumPy array:
t = torch.Tensor([[1, 2, 3], [3, 5, 6], [7, 8, 9]])
print("Pytorch tensor: ", t)
print("\n",type(t))
n = t.numpy()
print("\nNumpy array: ", n)
print("\n",type(n))
# Output:
Pytorch tensor:
tensor([[1., 2., 3.],
[3., 5., 6.],
[7., 8., 9.]])
<class 'torch.Tensor'>
Numpy array:
[[1. 2. 3.]
[3. 5. 6.]
[7. 8. 9.]]
<class 'numpy.ndarray'>
2. NumPy array to PyTorch tensor:
numpy_array = np.ones(5) # this method creates a numpy array with all elements as 1
print(numpy_array)
print(type(numpy_array))
pytorch_tensor = torch.from_numpy(n)
print("\n",pytorch_tensor)
print(type(pytorch_tensor))
# Output:
[1. 1. 1. 1. 1.]
<class 'numpy.ndarray'>
tensor([1., 1., 1., 1., 1.], dtype=torch.float64)
<class 'torch.Tensor'>
GPU/CPU compatibility:
If you have GPU on your computer and have also installed the CUDA toolkit for deep learning, then you can unleash the power of GPU to enable faster computation as compared to CPU.
tensor_1 = torch.Tensor([[1, 2, 3], [3, 5, 6], [7, 8, 9]])
tensor_2 = torch.Tensor([[10, 11, 12],[13, 14, 15],[16, 17, 18]])
if torch.cuda.is_available():
tensor_1 = tensor_1.cuda()
tensor_2 = tensor_2.cuda()
That’s all for today!
I hope you enjoyed learning the basics of PyTorch tensors. In the next tutorial, we will learn more about the datasets and dataloaders in PyTorch so that you can have fun handling data using PyTorch.
Article schedule
- Tensors
- Data handling using PyTorch
- Transforms
- Building a model using PyTorch
- Automatic differentiation
- Optimization loop
- Saving, loading, and using the saved model
Thanks for reading!
Bye!