Experiments with PyTorch and LeNet

In the PyTorch Neural Net tutorial, the article introduces LeNet — a convolution neural net consisting of two sets of convolution and average pooling layers and fully connected layers at the end. In the PyTorch Classifier tutorial, the article modifies the LeNet architecture to have an input of 3 channels and trains the network to classify pictures in the CIFAR-10 dataset. My objective was to experiment with the code given in the classifier tutorial and observe how LeNet’s changes its behavior.

Increasing Training Batch Size Decreases Accuracy

Increasing Batch Size decreases Accuracy

In the original code, the batch size for the “trainloader” and “testloader” was set to 4. The classifier had an accuracy of nearly 53% when I ran the code on my laptop. Now, for both the variables, I began increasing the batch size all the way up to 75. And, you can notice the drop in accuracy of the classifier in the picture above.

My first reaction was that probably this was due to memory issues. I was running PyTorch on a CPU and not a GPU. Anyways, I looked online for why this was happening. I ran into this post where the user had cranked up the batch size of a similar PyTorch LeNet implementation to 4096 on a GPU and found the accuracy of the classifier to drop to 19% for the CIFAR-10 dataset. The user reasoned that increasing batch size decreases the number of backpropagations and thus there are fewer updates to the classifier’s weights. Here’s another post where a user shared a paper that claims that for convolution neural networks large batch sizes “tend to converge to sharp minimizers of the training function”. In contrast, smaller batch sizes “tend to converge to flat minimizers”.

After reading these posts, I increased the batch size for the “testloader” to 100 but set the batch size for the “trainloader” at the original number 4. This time, I get back the usual accuracy of 53%. Thus, indeed, increasing batch size for the training set does affect LeNet’s behavior.

Varying Learning Rate

Accuracy % versus Learning Rate

The default optimizer for this classifier was SGD with a learning rate of 0.001. I tweaked the learning rate around to see if I got better results with a different value. Lowering the learning rate to 0.0007 gave similar accuracy. However, for all the other values that I tried, the classifier’s accuracy decreased.

Adam Optimizer

Since the default optimizer was SGD, I decided to check the accuracy of the convolution net with the Adam optimizer. So I changed the optimizer from SGD to Adam and set the learning rate to the usual 0.001 and I got an accuracy of 55%! In contrast, the standard accuracy of the classifier with SGD was 53%.

ConvNet Surgery

This was probably the most fun part of my experiments. I am always in awe of Machine Learning practitioners who pull out and add their own layers to a convolution neural net architecture. So I thought, well, I’ll give a small try myself.

For starters, I wanted to add just a padding layer before the input image went through the convolution layers. I had to get my math correct so that the dimension of the output layer matched up before it went through the fully connected layers.

So my modified process was:

• Input image of CIFAR-10 data has dimension 3 x 32 x 32
• Add a Zero Padding of 2. This changes dimension to 3 x 34 x 34. This layer is not present in the original article
• Pass The input through a Convolution Layer that has a kernel of 5 and output channel of 6. This layer spits out an output of 6 x 32 x 32
• Pass this through a Pooling Layer. The output dimension would be 6 x 16 x 16
• Pass this through a Convolution Layer that has a kernel of 5 and output channel of 16. The output layer dimension would be 16 x 12 x 12
• Pass this through a Pooling Layer again. The output dimension would be 16 x 6 x 6
• Pass the output layers through the fully connected layers
• I used Adam optimizer instead of SGD with momentum as I was getting better results with Adam

Here’s the code for the modified neural net and its forward pass:

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.pad  = nn.ZeroPad2d(2) #New Layer
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 6 * 6, 120) #Changed dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    def forward(self,x):
        x = self.pad(x) # New Padding Layer
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 6 * 6) #Changed dimension
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

With these modifications, I got an accuracy of 58%! For comparison, using the default code, the classifier’s accuracy was 53%. Small victories!