Dropout

What is Dropout Layer ?

Dropout layer randomly sets input units to 0 with a frequency of drop_rate at each iteration during training time, which help prevent overfitting.  

The key idea of dropout is randomly drop nodes along with their connection from the neural network during training time.

Dropout layer takes single float values as input between 0 to 1. In Keras implementation it denotes drop probability of unit. We will call it p_drop, so keep probability of unit is p_keep = 1 - p_drop.

Each unit is retained with a fixed probability (p_keep) independent of the other units. Generally we use p_drop 0.5 for dense layer.

Why do we need Dropout ? 

To solve the problem of Overfitting.

Overfitting means our model is performing well on training data but not performing well on test data (or new data).

One of the reason for overfitting is because our model is quite complex (having large number of parameter), so instead of just learning (generalizing) patterns/features in the data it also learn the noise present in the data and so it adjust it's weight to perform well on training data or we can also say that it adjust it's weight to memories the training data. And other reason of overfitting is training data is not good representation of overall (real) data. 

If training dataset is good representation of real data but not in good amount, this can also cause overfitting.

How dropout is solving the problem of Overfitting.

Multiple way to look into this

1. One way to look into this is, it reduces model complexity by randomly setting layer units to zero and so reducing model complexity that help in solving the overfitting.

2. During each training step, it drops unit with p_drop probability from the layer and then train a thinned network. 

Because at each training step, it trains a unique thinned network with less neurons, so the neuron present in network learn the representation(features) required for correct prediction. This prevent neurons from co-adapting too much on each other.

This make the network capable of better generalization and hence solving overfitting.

3. ""Overfitting can also be solved by training all possible neural network for a dataset and average the prediction form all model. But this is not possible.""

Let's see how we can interpret the above concept with dropout layer.

During training with dropout we train multiple sparse (thinned) neural network and at test time, we approximate the effect of averaging the predictions from all these thinned network by simply using original unthinned network that has smaller weights. This help in solving overfitting problem. Let's see in detail.

A neural network with n units, can be seen as 2^n possible thinned neural network and all these network shares the same weights.

During each training step we sample one out of 2^n network and train, so during whole training process we train multiple thinned network.

So training a neural network with dropout can be seen as training a collection of 2^n thinned network with extensive weight sharing, where each thinned network get trained very rarely, if at all. 

At test time, we can not take average of the prediction from all those networks. However simple approximate average method work well. So during inference time, idea is to use full network with all units with scaled-down version of weights. 

If a unit is retained with p_keep during training, then outgoing weights of that unit are multiplied by p_keep at test time. This ensure the expected out of hidden unit is same as the actual output at test time. By doing this scaling, 2^n network with shared weights can be combined into a single neural network to be used at test time.

 

Dropout during training and inference time

Lets say we want to apply dropout on this input data d = {1,2,3,4,5} with p_drop = 0.2 so now during training any one unit of d will become zero and d could be {1, 2, 3, 0, 5} because p_drop is 0.2 another way to look into this is we keep each node with probability (p_keep) 0.8 .

During inference time we will be using all the unit as dropout don't remove units during inference time. If we use all unit during inference, expected output will be different than training time. To make sure that the distribution of the values after the transformation during inference time remains almost the same, we multiply input with keep probability p_keep(1-p_drop) at inference time, during inference same d would be set to {0.8, 1.6, 2.4, 3.2, 4.0}. 

But in general we don't want to do anything with dropout layer during inference time so during training time only, we scale the values by 1/p_keep.

So now during training d could be set to {1.25, 2.5, 3.75, 0, 6.25} and nothing will happen with the input d during inference time.

That is why if you see Keras documentation of dropout it will say, dropout first set units to 0 with given drop probability(p) and then scale the remaining values by 1/(1-p).

That's all about dropout, thanks for reading the blog!

References

1. https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf

2. https://keras.io/api/layers/regularization_layers/dropout/

3. https://leimao.github.io/blog/Dropout-Explained/

Faster R-CNN : Object Detection

In this blog I will talk about Faster R-CNN algorithm. This is the third blog in the series of R-CNN based object detection. In my previous blogs I talked about R-CNN and Fast R-CNN, for better understanding of Faster R-CNN, first read about R-CNN and Fast R-CNN. Faster R-CNN is one of the state-of-the-art deep learning based object detection network, all the new object detection algorithm is compared with this one, so one must know about this.

Lets recall the previous object detection algorithm of the R-CNN family before starting about Faster R-CNN.

R-CNN (Region based convolutional neural network) have three component, first is region proposal using Selective Search algorithm that generate approx 2000 candidate regions for objects in an image, second part is feature extraction for all these candidate regions by passing them to CNN and third part is region classification using SVM and bounding box (b-box) refinement using b-box regression layer. 

R-CNN Object Detection

Fast R-CNN also have region proposal (Selective Search) same as R-CNN but instead of processing all region proposal through CNN, It first generate feature map of image by processing it by CNN and then extract features for each proposal from the feature map using ROI Pooling Layer.  

And these features are feed into a sequence of fully connected (FC) layers that finally branch into two sibling output layer, one produce softmax probability estimate over K+1 object classes (where K is no. of class, plus one for background) and other layer outputs four real-value number for each of the K classes. 

Fast-RCNN Object Detection

Fast R-CNN if fast and accurate in comparison to R-CNN but still it is not end-to-end trainable and region proposal is still bottelneck in state-of-the-art detection system.

Faster R-CNN introduces novel Region Proposal Networks (RPN) that share convolutional layers with detection networks. By sharing convolutions, the marginal cost for computing proposal reduces to 10ms from 1.5sec for an image. 

Faster R-CNN can be seen as RPN network plus Fast R-CNN detector network.

Let's try to understand RPN Network.

Region Proposal Network 

RPN network is going to generate candidate region for objects in the image, so it must generate proposal for object of different scale and aspect ratio.

To incorporate multiple scales and sizes, we have different scheme like pyramid of images and pyramid of filters. In pyramid of images, features map are calculated over multiple scale of images to detect smaller and larger object. In pyramid of filters, multiple filter with different scales/sizes are applied over feature map to detect smaller and larger object.  

Fig. (a) Pyramid of Image (b) Pyramid of Filters

RPN uses different scheme i.e. pyramid of reference boxes (Anchor). 

Fig. Pyramid of Reference Anchor

Anchor 

Anchor are pre-define bounding box of fixed shape and size. Anchor dimension are define wrt to images. For Anchors, Faster R-CNN uses 3 different scale with box area 128², 256² and 512² and 3 different aspect ration 1:1, 1:2 and 2:1, combination of these will produce 9 anchors.

All 9 anchor at pixel (320,320). Area for blue, green, and red anchors are 128², 256² and 512² respectively.

For a typical input image 1000x600x3 in Faster R-CNN, the dimension of feature map is reduced to 60x40x512 after backbone (VGG-16) network. This feature map is shared between RPN network and Fast R-CNN detection network.

RPN add a mini-convolutional network over backbone/head network to output a set of rectangular object proposal. This mini-network slide multiple 3x3 filter on the last convolutional layer of backbone network, followed by, two sibling 1x1 convolutional layer, a box-regression layer and box-classification layer for object proposal prediction.

At each sliding window (convolution) location it predict 9 proposal (b-box) of different scale and size using Anchors.

Region Proposal Network

The first conv layer of RPN help the RPN network to learn feature for anchors prediction on top of the base network features. The first 1x1 convolution branch of RPN predict whether proposal containing object or no-object. So the number of conv filter required to classify one anchor is 2 and for 9 anchor is 18. 

The Second 1x1 convolution branch of RPN predict proposal bounding box offset. So the number of conv filter required to predict b-box offset for one anchor is 4 (x,y,w,h) and for 9 anchor is 36.

For VGG and ZF head network, network stride is 16 which means after processing image by any of these head network, feature map dimension will be 1/16 of input image dimension. One pixel in feature map have 16x16 receptive field in the input image. So when we say at each point of feature map we try to detect 9 different proposal that means we try to locate object in input image after every 16 pixels.

To understand how RPN learn to predict the object location, see it's loss function. 

RPN Loss function

RPN network is trained in a batch of 256 anchor, with 1:1 ratio of positive anchor and negative anchor. Positive anchor have IoU grater than 0.7 with ground truth and negative anchor have IoU less than 0.3 with ground truth, other anchor are defined as neutral anchor. Note one ground truth can have IoU of 0.7 with multiple anchor.

RPN loss have two component, b-box regression loss to predict the location of object and b-box classification loss to predict positive object and negative object. Regression loss is only calculated for positive anchors.

Multi task loss of RPN

Here, i the index of an anchor in a mini-batch.

First part of above loss is classification loss where p_i is the predicted probability of anchor i being an object and p^*_i is the ground truth of the anchor i, here p^*_i is 1 for positive anchor and 0 for negative anchor. This classification loss L_cls is log loss over two classes (object vs not object).

Second part of loss is regression loss which is multiplied by p^*_i so it's value for negative anchor is zero, here t_i is a vector representing the 4 parameterized co-ordinates of the predicted b-box and t^*_i is that of the ground-truth box associated with a positive anchor.

For Regression loss, L_reg(t_i, t^*_i), it uses smooth L1 loss. To read more about smooth L1 loss go through the loss section of previous blog.

Classification loss is normalized by mini batch size N_cls (256), regression loss is normalized by number of anchor location N_reg (~2400) and regression loss is weighted by lambda which is 10 thus giving approximately equal weighted to both loss.

The regression target t^*_i for a positive anchor is defined as -

Offset of ground truth box and anchor

Here x, y, w and h denotes the box's center coordinates and it's width and height.

Variable x, x_a and x^* are for predicted, anchor and ground truth respectively, likewise for y, w and h.

Predicted bounding box parameter (x, y, w, h) can be calculated by using predicted offset and corresponding anchor.

Offset of predicted box and anchor

RPN loss function force the RPN network to learn to predict the offset (t_x, t_y, t_w and t_h) of bounding box wrt pre-define anchors for object. 

We can see this as a bounding box regression from an anchor box to a nearby ground-truth box.

Training RPN

RPN can be trained end-to-end using backpropogation and SGD. Weights of backbone network is initialized by pre-trained model for ImageNet classification. New layers weight are randomly initialized using zero mean Gaussian distribution with standard deviation of 0.01.

To compute the loss of RPN, random sample of 256 anchor are selected from an image with 1:1 ration of positive and negative anchor. In case of less (<128) positive anchor in an image mini batch is padded with negative anchor.

The Anchor boxes that cross image boundaries were removed from training which reduces the number of anchor from ~20000 (60x40x9) to 6000. Some proposals highly overlap with each other so to reduce redundancy NMS is used with 0.7 IoU threshold, which leaves around 2000 proposal regions per image. After NMS top-N proposal are used for detection.

Faster R-CNN : RPN + Fast R-CNN detector

Faster R-CNN network can be seen as Fast R-CNN network with RPN network for object proposal network instead of Selective Search Algorithm. 

Faster R-CNN Network

Backbone/Head network (VGG-16) feature map is shared between both RPN and detection branch.

Detection network project the object proposal from RPN network to backbone network feature map to get the features for object classification and object bounding box refinement but these feature need to be a fixed size because of fully connected layer in network.

So it uses RoI pooling layer to resize the feature map of each object to a fixed size (7x7x512).

RoI pooling works by dividing the H x W roi region (object proposal) into h x w grid of sub-window of approximate size H/h x W/w and then max-pooling the values in each sub-window into the corresponding output grid cell. Read more about RoI pooling layer in Fast R-CNN blog.

After RoI pooling layer, resized feature for each proposal is passed to fully connected layer and that feature is finally passed to classification layer and bounding box refinement layer.

Classification layer gives C probability values using Softmax function for each proposal where C is number of class including background. 

Predicted box of object proposal is further refined by bounding box regression layer of detection network. This layer gives bounding box offset wrt to each class. That means each class have their own regression with four parameter unlike bounding box regression of RPN.

It uses the same multi-task loss as in Fast R-CNN.

One of the important thing to note about this network is feature sharing between RPN and detection network.

Feature Sharing for RPN and Fast R-CNN

If both RPN and detector network trained independently, both will modify backbone convolutional layer weights in different ways. So author followed a 4-step training procedure to allow the networks to share the weights.

1. First, RPN is trained independently as mentioned above. The network initialized with ImageNet pre-trained model and fine-tuned end-to-end for the region proposal task.
2. In second step, separate detection network is trained by using the proposal generated by step-1 RPN. Again this network is also initialized with ImageNet pre-trained model. Till here networks are not sharing weights.
3. In third step, RPN is again trained but this time network is initialized with the above step-2 detector weights and keeping the common convolutional layer weights between RPN and detector fixed, only layer unique to RPN are fine-tuned. Now here both network are sharing the same wights for backbone network.
4. Finally, keeping the shared convolutional layers fixed of above network, layers unique to detection branch (Faster R-CNN) are fine-tuned. This gives the final Faster R-CNN network which share the convolutional weight and form a unified network.

Result

Faster R-CNN using VGG-16 as backbone network achieves state of the art object detection accuracy on PASCAL VOC 2007, 2012 and MS COCO dataset with only 300 proposal. It perform at 5 fps including all step on a GPU.

• Faster R-CNN takes around 198ms for proposal and detection on one image compared to approx 1.8 sec for Fast R-CNN (avg 1.5 sec for proposal and 320ms for detection).
• Faster R-CNN achieved 3.2% higher mAP compared to Fast R-CNN on the union set of PASCAL VOC 2007 trainval and 2012 trainval dataset.
• Faster R-CNN achieved 2.8% higher mAP@0.5 IoU compared to Fast R-CNN on MS COCO dataset.

That's all for this post, hope you find this blog informative.
Thanks for reading !!

Image Classification in Keras

In this blog I will explain how to do image classification in python using Keras. Image classification is a basic problem in Deep Learning. It is a method to classify images into their respective classes.

image_1.jpg

image_2.jpg

Here is an example of Image classification, image_1.jpg should be predicted as cat and image_2.jpg should be predicted as dog by CNN (Convolutional Neural Network) model.

Keras gives us lots of CNN (VGG, ResNet, Inception etc.) models already trained on ImageNet Dataset for 1000 classes (http://www.image-net.org/), we will be using VGG-16 model and fine-tune it on dog-vs-cat dataset (https://www.kaggle.com/c/dogs-vs-cats/data) for two classes.

Data Preparation 

How to prepare Train, Validation and Test dataset ?

Train dataset is the only dataset that is used for model training (learning weights and biases) and validation dataset is used to monitor the loss and accuracy of intermediate model on unseen data. Train and validation dataset is used during training only.

After training we can choose top three model which is performing good on validation data, and test these models on the test dataset to select the best model.

We will use 80% (8000) of the total data (10000) for training purpose and 10%-10% (1000-1000) for validation and testing. It’s not a rule to use 10% of the whole dataset for testing and validation, sometimes we also use 1-2% of total images when we have millions of images.

We need three directory train, validation & test. Each directory should contain one sub-directory for each class filled with respective images.

-- train/

     -- cat/

     --dog/

--validation/

     --cat/

     --dog/

--test/

     --cat/

     --dog/

Code and Explanation

In this section i will explain code and important points regarding classification model.

First let's import all the required packages and modules.

As VGG-16 model is trained on 224x224x3 image size so we will be using same size for fine tuning however we can also fine-tune it for different size like 100x100x3 or 448x448x3, we will see how to fine-tune model for different image size in my future blogs.

As we want to modify the VGG-16 model for our own dataset we will load only convolution blocks not fully connected and output layers. This line will load the VGG-16 model with ImageNet weights without fully connected and output layers as we set “include_top” argument as false.

Original VGG-16 have two fully connected layers with 4096 neurons and output layer for 1000 classes.

As cat and dog images are not very difficult to differentiate so we will be using only 256 neurons in fully connected (FC) layer instead of 4096 neurons as in original VGG-16 network. Final prediction layer will have only one neuron for binary classification.

You can see summary of our network using model.summary() function.

Freeze Layer

Freezing any layer means we won't be modifying the weights and biases of that layer during training. This is one of the important things to do while fine tuning any already pre-trained model.

Why to freeze layers ?

As we know each layer in any network learns to detect some kind of feature like different kind of edges, corners, patches, color and template like feature from images so during fine tuning pre-trained model we can use some of the features (like edge and corner features) as it is useful for all problem statement. That's why we can freeze some layers to use already learned features.

If we freeze some layers of network that means we are modifying less layers during training process, so training and convergence process of model should be fast.

When to freeze the layers of the base model ??

One can freeze some layers during fine-tuning of pre-trained model in case of

• If you have less training images
• If new classes are similar to old classes on which model is already trained, and if new classes are totally different then one would have to train more layers of the base network like in case of fine-tuning VGG model on medical images.

Currently we are training only 30% layers of base model, If we have more images we can train more layers.

Note: Layer freezing of model should be done before model compilation.

Now let's compile the model with SGD optimizer and binary crossentropy loss. For more than two classes use categorical_crossentropy.

One can use Adam optimizer or other optimizer because some time some optimizer perform well on some data and the same optimizer with same parameters doesn’t perform well on different types of data. So it’s always better to experiment with two-three optimizers and select the one which performs well on your data.

Data Loading

Let's see how to load the data and perform augmentation using ImageDataGenerator class of Keras.

At training and validation step we want to normalize the data and at training time we also want to augment the images to increase the training dataset size. Here we are using shear, zoom and horizontal flip techniques to augment more training images.

flow_from_directory is the function that load the images from given directory, we have to pass the path of the directory containing sub-directory for each classes. Here we are using 'binary' class_mode because we have only two classes, for more than two classes use 'categorical' class_mode.

Callback

A callback is a set of functions to be applied at given stages of the training procedure. With the help of callback functions we can monitor the loss, save weights files and plot training & validation loss graphs during training.

To make things simple we will be using only ModelCheckpoint callback for now. With this callback we can save weights when their is increment in validation accuracy or decrement in validation loss controlled by monitor and mode parameter.

Enabling save_best_only parameter will save weight file if new weights are better than the previous saved weight, disabling it will save weight file of all epochs.

save_weights_only is the important parameter, if it’s true then it will save only weights of the network not the network architecture. To use this weights you have to first create the network then you can load the weights with load_weights function.

If save_weights_only is false then while saving weights it will save network architecture, training configuration and state of optimizer which allow you to resume the training where you left off. You don’t need network architecture information, you can directly load weights and network by using load_model function.

By default save_weights_only is False.

Training

Now we will use fit_generator function to start the training.

There are few terms that one should know.

Epoch - While training any deep learning model, the model is training on the whole dataset many times not just once. Epoch defined as one pass over the entire dataset. Model trained for one epoch means models have seen the whole training images once.

Batch Size - A set of N images. The samples in a batch are processed independently, in parallel. During training weights are going to be updated after each batch. Batch size can be 32, 64, 128 or 256 , depending on GPU and memory size of your system use appropriate batch size. Batch size also affects the convergence of the model I will not go in detail of this.

Steps per epoch - steps_per_epoch * batch_size = total training samples

So steps per epoch should be equal to total_training_sample divided by batch_size if you set less than the above calculated number then you will not use all training images during one epoch.

Let's train this network for 25 epoch.

Here is training summary till 8th epoch. Model achieved 98% accuracy in 6th epoch itself.

Inference
Let's see how to do inference on test images.

 Import libraries to load model and to read images.

Here we have defined the classes and loaded the model using load_model function.

In the above code we have loaded the image using PIL image processing library (I have used PIL library because Keras data loader internally uses PIL to read the images at training time), and resized the image to (224,224,3) as our model is trained for this size and then we have added one more dimension in image to make image dimension from (224,224,3) to (1,224,224,3). Model accept four dimension input, added dimension is to represent batch_size. And finally we have normalized the image by dividing it by 255.

Predict function on image for binary classification give us the probability between 0 & 1. Put a threshold on probability if less then threshold then class 0 else class 1 and here we have used 0.5 as threshold value. If model is biased toward one class then you might want to change the threshold value.

So in this blog we have seen how to fine tune VGG-16 model for new dataset & achieve good accuracy and how to use new trained model for inference on images.

That's all for this blog, hope you find this blog informative.
Thanks for reading !!

Code, used dataset  and model link are below.

Full code github link - https://github.com/111surajmaurya/VGG-16

Dataset google drive link - https://drive.google.com/drive/folders/15Et639VosTWxEmPLRqx4hdTdGvzQupbg?usp=sharing

Model link - https://drive.google.com/drive/folders/1F72YB-D4XOlIvnujE--HPj9lMUkXsjKe?usp=sharing