In my journey to become fluent in writing Machine Learning software, this project represents the second stage of my development. I started by creating a Neural Network from scratch in C++ to recognise handwritten digits, you can find it in its repository Machine Learning C++. Here, I start experimenting with PyTorch through Python.
The datasets used in this repository for training, with increasing difficulty, are the MNIST, the Fashion MNIST, and the CIFAR10. For each dataset, I created a separate folder containing the training script, the best trained weights, and a showcase file to visualize real-time predictions.
One of the most important things to figure out was how to build a proper Machine Learning file that could handle all the details on its own while I could just worry about the Neural Network design and tuning. The way all the different projects treat the training is as follows:
-
Configuration data-class: All the files have a data-class called
Configwhere all the hyperparameters can be tweaked, making the tuning part of the machine learning process more accessible. -
Neural Network class: Each project has its own definition for the Neural Network they use, it has a initializer where the different blocks are declared. A forward function that performs the forward pass using the initialized blocks, and returns the logits and also the probabilities if requested. And also a load function to load previously trained weights from a file.
-
Data pipeline: The datasets are fully loaded into RAM before training to avoid I/O bottlenecks, and can either be stored in the CPU or straight in the GPU, during training dataloaders are used to apply the necessary augmentations to the stored inputs and send the batches to the forward pass.
-
Training Loop: The training functions iterate through the batches for a given amount of epochs until the training is finished. At the end of every epoch they evaluate the loss and accuracy on the test set, print the results to the console, and if the result is the best so far it gets stored.
Following with the same dataset as before was to see what improvements could be done using basic Convolutional Neural Networks. It turns out that Convolutional Neural Networks are very porweful for image classification and I immediately found myself past the 99% accuracy mark.
So far the best result with a small CNN has been 99.65% accuracy on the test set. The layout is two blocks with three convolutional layers each, the first full size with 16 channels and the second half the size with 32 channels. And at the end a small dense block.
self.conv_block1 = nn.Sequential(
nn.Conv2d(1, 16, kernel_size=5, padding=2, bias=False),
nn.BatchNorm2d(16),
nn.ReLU(inplace=True),
nn.Conv2d(16, 16, kernel_size=5, padding=2, bias=False),
nn.BatchNorm2d(16),
nn.ReLU(inplace=True),
nn.Conv2d(16, 16, kernel_size=5, padding=2, bias=False),
nn.BatchNorm2d(16),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2),
)
self.conv_block2 = nn.Sequential(
nn.Conv2d(16, 32, kernel_size=5, padding=2, bias=False),
nn.BatchNorm2d(32),
nn.ReLU(inplace=True),
nn.Conv2d(32, 32, kernel_size=5, padding=2, bias=False),
nn.BatchNorm2d(32),
nn.ReLU(inplace=True),
nn.Conv2d(32, 32, kernel_size=5, padding=1, bias=False),
nn.BatchNorm2d(32),
nn.ReLU(inplace=True),
nn.MaxPool2d(kernel_size=2, stride=2),
)
self.dense_block = nn.Sequential(
nn.Flatten(),
nn.Linear(36*32, 64),
nn.ReLU(inplace=True),
nn.Dropout(cfg.dropout_p),
nn.Linear(64, 32),
nn.ReLU(inplace=True),
nn.Dropout(cfg.dropout_p),
nn.Linear(32,10),
)The reasoning for the dense block at the end is quite simple, if you understand the convolutional layers as pattern finders, numbers are definitely made out of patterns but the position of those patterns is also important, so if you just feed an average pooling to a last linear layer you are loosing a lot of valuable spatial information, and the final prediction does not have the full picture. This layout proved to work really well and that is why it is the one standing so far.
For showcasing, I built what I consider the most fun app, this one instead of providing images and labels it allows the user to draw numbers itself, it is quite simple but very effective at seeing where the flaws are in a given Neural Network.
This image shows a screenshot of the app predicting an example in real time.
For the case of the other two datasets, image recognition proved to be a much more difficult task. After many scalings of the Neural Network there were some accuracy marks I could not get past. So the decision was made to implement more complex Convolutional Neural Networks to solve the problem.
As the layout for both datasets use different variants of a ResNet (v1) Neural Network were used. The idea underlying this design is that after every block, composed of two convolutional layers, the input is also added at the end, resulting in the possibility of blocks being inactive and barely making changes to a specific image, and the image making its way down the forward pass. This allows for strong specialization of every single block and provides better results.
These blocks are organized in three stages where the image is downscaled using max pooling and the channels are doubled using a convolutional layer of kernel size one.
The layout in code looks like this:
self.stem = nn.Sequential(
nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1, bias=False),
nn.BatchNorm2d(16),
nn.ReLU(inplace=True),
)
self.stage1 = Stage(N, 16, 16*K, downsize=False, dropout=dropout)
self.stage2 = Stage(N, 16*K, 32*K, downsize=True, dropout=dropout)
self.stage3 = Stage(N, 32*K, 64*K, downsize=True, dropout=dropout)
self.head = nn.Sequential(
nn.AvgPool2d(kernel_size=8, stride=8),
nn.Flatten(),
nn.Linear(64*K,10),
)Each stage contains N blocks with the specified number of channels multiplied by K, for my particular case due to computational limitation I did not use Wide ResNets, keeping the K value at one. The ResNet is labeled depending on the values of these two parameters.
In the particular cases covered in this repository ResNet-14 (v1) was used for the Fashion MNIST
dataset, achieving accuracies of up to ~96%. And ResNet-26 (v1) was used for the CIFAR10 dataset,
achieving accuracies up to ~90%. In both cases MixUp was used for data augmentation, mixing the
images between them to soften the predictions and significantly decrease the chances of overfitting.
Due to my hardware limitations, I couldn’t afford to train Wide ResNets, but with
a similar setup using a WRN-28-10 (v1), which uses 10 times more channels, a team of researchers
demonstrated state-of-the-art performance on CIFAR-10 and CIFAR-100, reaching accuracies above 96%
and 81% respectively.
For the showcase both datasets have a dedicated app that simply prints the test set images on the screen and allows the user to switch between them and view the predictions and also apply the augmentations as specified in the configuration.
This image shows a screenshot of both apps doing correct predictions on images from the test sets.
The current results are quite good for my standards and this project was never meant to be revolutionary, but rather a deeper dive into intermediate-level image recognition. For that task I consider it has been very productive.
It is time to move on to other branches of Machine Learning, potentially participating in some Kaggle competitions but I do not discard coming back to this project in the future with more knowledge (and a big GPU 😄) and going for my goal of cracking the CIFAR100.