Batch Normalization: Accelerating Deep Network Training

In this blog we will talk about Batch Normalization layer.

Before understanding Batch Normalization first we need to understand what is internal covariant shift. 

Internal Covariant Shift is defined as the change in the distribution of  the network activation (feature map) due to change in the network parameter during training. 

So because of this internal covariant shift, training deep neural network is difficult because hidden layers input distribution keep changing, so the layers need to be continuously adapt to the new distribution. 

Theta2 need to readjust to re-compensate for the change in the distribution of x. This slow down the training by requiring lower learning rate and careful parameter initialization.  

To address this problem we can normalize the layer inputs, that's where batch normalization is useful.

Why we call this a Batch normalization because, training happens over a batch (64/128/256) of training images so we normalize batch of feature maps.

For a layer with k-dimensional input x = ( x(1).......x(k)), we will normalize each dimension, where the expectation and variance are computer over the training data set. Such normalization speed up the convergence.

The resulting normalized activation x^(k) have zero mean and unit variance.

Note that simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the non linearity.

To address this, we make sure that the transformation inserted in the network can represent the identity transform. To accomplish this we introduce, for each activation x(k), a pair of parameters γ(k), β(k), which scale and shift the normalized value. 

These parameters are learned along with the original model parameters, and restore the representation power of the network.  

Indeed, by setting γ(k)=√Var[x(k)] and β(k)=E[x(k)], we could recover the original activations, if that were the optimal thing to do. 

Batch Normalization During Training

• During Training, mean and variance of the each sample of mini-batch is calculated and for each sample mean and variance of each dimension (channels) are calculated separately. 
  
  

   

   
• Then we normalize the input feature map using calculate mean and variance to make the distribution with zero mean and unit variance.
  
  
   
   
• As we are changing (normalizing) the input of layer, this may change what layer can represent. To address this, we make sure that the transformation inserted in the network can represent the identity transform by using γ (scale) and β (shift) parameters.
  
  
  
  

Batch Norm during training - 

Batch Normalization During Inference

• The normalization of activations that depends on the mini-batch allows efficient training, but is neither necessary nor desirable during inference, we want the output to depend only on the input, deterministically.

• Instead of using mini-batch statistic we use population statistics that is moving average mean and variance of whole mini batch used in training. Moving mean and moving variance are non trainable parameters that are  calculated (updated) and stored during training.

  • moving_mean(t) = moving_mean(t-1)* momentum + mean(batch)(t) * (1 - momentum)
  • moving_var(t) = moving_var(t-1) * momentum + var(batch)(t) * (1 - momentum)
  • momentum = 0.99


• Then normalize the testing sample using population mean and variance. After that we need to transform this normalized sample using learned scale (γ) and shift (β) parameter of the respective layer to yield single linear transformation.

• Since the parameters are fixed in this transformation, the batch normalization procedure is essentially applying a linear transform to the activation.
  

One more important question for batch normalization is after which layer batch normalization should be used before activation layer or after activation layer.

• We can use batch normalization layer before or after activation layer
  

• For Sigmoid and Hyperbolic tangent (Tanh), s-shaped function, one can use batch normalization after activation function

• For activation which may result in non-Gaussian distribution like rectified linear activation function, one can use batch normalization before activation function

• Batch Normalization Paper suggests, Batch Normalization to use  before Activation Function
  

Let's see the output of batch normalization layers (Keras implementation), after a convolution layer whose output feature map dimension is (batch_size, 14, 14, 128). 

Observe the dimension of γ, β, moving mean and moving variance is equal to number of channel of input feature map.

That's all about Batch Normalization.

Thanks for reading the blog.

Reference

• https://arxiv.org/pdf/1502.03167.pdf
• https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/
  

Faster R-CNN : Object Detection

In this blog I will talk about Faster R-CNN algorithm. This is the third blog in the series of R-CNN based object detection. In my previous blogs I talked about R-CNN and Fast R-CNN, for better understanding of Faster R-CNN, first read about R-CNN and Fast R-CNN. Faster R-CNN is one of the state-of-the-art deep learning based object detection network, all the new object detection algorithm is compared with this one, so one must know about this.

Lets recall the previous object detection algorithm of the R-CNN family before starting about Faster R-CNN.

R-CNN (Region based convolutional neural network) have three component, first is region proposal using Selective Search algorithm that generate approx 2000 candidate regions for objects in an image, second part is feature extraction for all these candidate regions by passing them to CNN and third part is region classification using SVM and bounding box (b-box) refinement using b-box regression layer. 

R-CNN Object Detection

Fast R-CNN also have region proposal (Selective Search) same as R-CNN but instead of processing all region proposal through CNN, It first generate feature map of image by processing it by CNN and then extract features for each proposal from the feature map using ROI Pooling Layer.  

And these features are feed into a sequence of fully connected (FC) layers that finally branch into two sibling output layer, one produce softmax probability estimate over K+1 object classes (where K is no. of class, plus one for background) and other layer outputs four real-value number for each of the K classes. 

Fast-RCNN Object Detection

Fast R-CNN if fast and accurate in comparison to R-CNN but still it is not end-to-end trainable and region proposal is still bottelneck in state-of-the-art detection system.

Faster R-CNN introduces novel Region Proposal Networks (RPN) that share convolutional layers with detection networks. By sharing convolutions, the marginal cost for computing proposal reduces to 10ms from 1.5sec for an image. 

Faster R-CNN can be seen as RPN network plus Fast R-CNN detector network.

Let's try to understand RPN Network.

Region Proposal Network 

RPN network is going to generate candidate region for objects in the image, so it must generate proposal for object of different scale and aspect ratio.

To incorporate multiple scales and sizes, we have different scheme like pyramid of images and pyramid of filters. In pyramid of images, features map are calculated over multiple scale of images to detect smaller and larger object. In pyramid of filters, multiple filter with different scales/sizes are applied over feature map to detect smaller and larger object.  

Fig. (a) Pyramid of Image (b) Pyramid of Filters

RPN uses different scheme i.e. pyramid of reference boxes (Anchor). 

Fig. Pyramid of Reference Anchor

Anchor 

Anchor are pre-define bounding box of fixed shape and size. Anchor dimension are define wrt to images. For Anchors, Faster R-CNN uses 3 different scale with box area 128², 256² and 512² and 3 different aspect ration 1:1, 1:2 and 2:1, combination of these will produce 9 anchors.

All 9 anchor at pixel (320,320). Area for blue, green, and red anchors are 128², 256² and 512² respectively.

For a typical input image 1000x600x3 in Faster R-CNN, the dimension of feature map is reduced to 60x40x512 after backbone (VGG-16) network. This feature map is shared between RPN network and Fast R-CNN detection network.

RPN add a mini-convolutional network over backbone/head network to output a set of rectangular object proposal. This mini-network slide multiple 3x3 filter on the last convolutional layer of backbone network, followed by, two sibling 1x1 convolutional layer, a box-regression layer and box-classification layer for object proposal prediction.

At each sliding window (convolution) location it predict 9 proposal (b-box) of different scale and size using Anchors.

Region Proposal Network

The first conv layer of RPN help the RPN network to learn feature for anchors prediction on top of the base network features. The first 1x1 convolution branch of RPN predict whether proposal containing object or no-object. So the number of conv filter required to classify one anchor is 2 and for 9 anchor is 18. 

The Second 1x1 convolution branch of RPN predict proposal bounding box offset. So the number of conv filter required to predict b-box offset for one anchor is 4 (x,y,w,h) and for 9 anchor is 36.

For VGG and ZF head network, network stride is 16 which means after processing image by any of these head network, feature map dimension will be 1/16 of input image dimension. One pixel in feature map have 16x16 receptive field in the input image. So when we say at each point of feature map we try to detect 9 different proposal that means we try to locate object in input image after every 16 pixels.

To understand how RPN learn to predict the object location, see it's loss function. 

RPN Loss function

RPN network is trained in a batch of 256 anchor, with 1:1 ratio of positive anchor and negative anchor. Positive anchor have IoU grater than 0.7 with ground truth and negative anchor have IoU less than 0.3 with ground truth, other anchor are defined as neutral anchor. Note one ground truth can have IoU of 0.7 with multiple anchor.

RPN loss have two component, b-box regression loss to predict the location of object and b-box classification loss to predict positive object and negative object. Regression loss is only calculated for positive anchors.

Multi task loss of RPN

Here, i the index of an anchor in a mini-batch.

First part of above loss is classification loss where p_i is the predicted probability of anchor i being an object and p^*_i is the ground truth of the anchor i, here p^*_i is 1 for positive anchor and 0 for negative anchor. This classification loss L_cls is log loss over two classes (object vs not object).

Second part of loss is regression loss which is multiplied by p^*_i so it's value for negative anchor is zero, here t_i is a vector representing the 4 parameterized co-ordinates of the predicted b-box and t^*_i is that of the ground-truth box associated with a positive anchor.

For Regression loss, L_reg(t_i, t^*_i), it uses smooth L1 loss. To read more about smooth L1 loss go through the loss section of previous blog.

Classification loss is normalized by mini batch size N_cls (256), regression loss is normalized by number of anchor location N_reg (~2400) and regression loss is weighted by lambda which is 10 thus giving approximately equal weighted to both loss.

The regression target t^*_i for a positive anchor is defined as -

Offset of ground truth box and anchor

Here x, y, w and h denotes the box's center coordinates and it's width and height.

Variable x, x_a and x^* are for predicted, anchor and ground truth respectively, likewise for y, w and h.

Predicted bounding box parameter (x, y, w, h) can be calculated by using predicted offset and corresponding anchor.

Offset of predicted box and anchor

RPN loss function force the RPN network to learn to predict the offset (t_x, t_y, t_w and t_h) of bounding box wrt pre-define anchors for object. 

We can see this as a bounding box regression from an anchor box to a nearby ground-truth box.

Training RPN

RPN can be trained end-to-end using backpropogation and SGD. Weights of backbone network is initialized by pre-trained model for ImageNet classification. New layers weight are randomly initialized using zero mean Gaussian distribution with standard deviation of 0.01.

To compute the loss of RPN, random sample of 256 anchor are selected from an image with 1:1 ration of positive and negative anchor. In case of less (<128) positive anchor in an image mini batch is padded with negative anchor.

The Anchor boxes that cross image boundaries were removed from training which reduces the number of anchor from ~20000 (60x40x9) to 6000. Some proposals highly overlap with each other so to reduce redundancy NMS is used with 0.7 IoU threshold, which leaves around 2000 proposal regions per image. After NMS top-N proposal are used for detection.

Faster R-CNN : RPN + Fast R-CNN detector

Faster R-CNN network can be seen as Fast R-CNN network with RPN network for object proposal network instead of Selective Search Algorithm. 

Faster R-CNN Network

Backbone/Head network (VGG-16) feature map is shared between both RPN and detection branch.

Detection network project the object proposal from RPN network to backbone network feature map to get the features for object classification and object bounding box refinement but these feature need to be a fixed size because of fully connected layer in network.

So it uses RoI pooling layer to resize the feature map of each object to a fixed size (7x7x512).

RoI pooling works by dividing the H x W roi region (object proposal) into h x w grid of sub-window of approximate size H/h x W/w and then max-pooling the values in each sub-window into the corresponding output grid cell. Read more about RoI pooling layer in Fast R-CNN blog.

After RoI pooling layer, resized feature for each proposal is passed to fully connected layer and that feature is finally passed to classification layer and bounding box refinement layer.

Classification layer gives C probability values using Softmax function for each proposal where C is number of class including background. 

Predicted box of object proposal is further refined by bounding box regression layer of detection network. This layer gives bounding box offset wrt to each class. That means each class have their own regression with four parameter unlike bounding box regression of RPN.

It uses the same multi-task loss as in Fast R-CNN.

One of the important thing to note about this network is feature sharing between RPN and detection network.

Feature Sharing for RPN and Fast R-CNN

If both RPN and detector network trained independently, both will modify backbone convolutional layer weights in different ways. So author followed a 4-step training procedure to allow the networks to share the weights.

1. First, RPN is trained independently as mentioned above. The network initialized with ImageNet pre-trained model and fine-tuned end-to-end for the region proposal task.
2. In second step, separate detection network is trained by using the proposal generated by step-1 RPN. Again this network is also initialized with ImageNet pre-trained model. Till here networks are not sharing weights.
3. In third step, RPN is again trained but this time network is initialized with the above step-2 detector weights and keeping the common convolutional layer weights between RPN and detector fixed, only layer unique to RPN are fine-tuned. Now here both network are sharing the same wights for backbone network.
4. Finally, keeping the shared convolutional layers fixed of above network, layers unique to detection branch (Faster R-CNN) are fine-tuned. This gives the final Faster R-CNN network which share the convolutional weight and form a unified network.

Result

Faster R-CNN using VGG-16 as backbone network achieves state of the art object detection accuracy on PASCAL VOC 2007, 2012 and MS COCO dataset with only 300 proposal. It perform at 5 fps including all step on a GPU.

• Faster R-CNN takes around 198ms for proposal and detection on one image compared to approx 1.8 sec for Fast R-CNN (avg 1.5 sec for proposal and 320ms for detection).
• Faster R-CNN achieved 3.2% higher mAP compared to Fast R-CNN on the union set of PASCAL VOC 2007 trainval and 2012 trainval dataset.
• Faster R-CNN achieved 2.8% higher mAP@0.5 IoU compared to Fast R-CNN on MS COCO dataset.

That's all for this post, hope you find this blog informative.
Thanks for reading !!

VGG-16 Inference with different image dimension

In this blog i will talk about how to create a classification network or fine-tune any pre-trained classification network (VGG-16) that accepts image of any dimension rather than one dimension on which it is trained. Generally a model trained for MxNx3 dimension images accept only MxNx3 input not other dimensions but here we will see how to modify network to accept different dimension input image also.

I will explain this with VGG-16 network.

(**Dimension is referred to only width and height of image/feature map not channel/depth) 

Let's talk about scenario where a single network accepting different dimensions input can be helpful.

• Case -1 You have few hundred images of smaller dimension (let's say 100x100x3) then fine-tuning a pre-trained network is one of the best option you have instead of training one from scratch.
• Case-2 You trained a classification network for a fixed input dimension (224x224x3) and at inference time you get different dimension input ( ranging from 200x200x3 to 300x300x3) and you don't want to pad or resize inference images to loose information.
• Case-3 You have multiple dataset with different input dimension then with this kind of network you can easily train a classifier without modifying input dataset.

There can be other scenario where it is helpful but let's not talk about all and move forward with solution.

So the first question is what's the problem if we pass different dimensions images to a VGG-16 network, which layer will create problem convolution or pooling or flatten or fully connect (FC) layer ??

Let's see what these layers do in brief.

• Convolution layer accept any dimension input and perform convolution with kernel values and its output dimensions depend on the padding or stride of kernel. It may or may not reduce dimension.
• Pooling layer accept any dimension input and its output dimension depend on stride of pooling operation. It will reduce dimension.
• Flatten layer accept any dimension input and its output is reshaped input in single dimension.
• Fully Connect (FC) layer accept fixed input dimension and its output dimension depend on next FC layer input dimension or output layer dimension i.e. both are fixed dimensions. It may or may not reduce dimension.

Now let's see network architecture for original VGG-16 which is trained on 224x224x3 images and same VGG-16 network when trained on 150x150x3 images.

For different dimensions of input images, after block5_pool layer feature map (feature map is nothing but the output of convolution or pooling layer of CNN ) dimensions is different because of convolution and pooling layer as they reduce feature map size by some constant factor and after that flatten layer is just flattening the feature map to one dimensional vector form.

We can see input to first FC layer is 25088 (7x7x512) when image is 224x224x3 and 8192 (4x4x512) when input is 150x150x3, so this will create a problem if you pass 150x150x3 image to a network which is trained for 224x224x3

In the above image you can see i have loaded original VGG-16 model for 1000 classes, and it gave error for 374x500x3 dimension input image, but if you uncomment resize line then it will run and give probability for 1000 classes.

So a network trained for 224x224x3 will take only 224x224x3 dimension input not 150x150x3 and vice versa. 

If we want a single network to accept both images then output dimension of flatten layer should be fixed so that FC layer should always accept the output of flatten layer. The problem will be solved if somehow we always pass fixed input dimension to FC layer And that's where Global Average Pooling Layer help us.

Conclusion till here is because FC layer accept fixed length input that's why passing different dimension image to VGG-16 network results in the error. 

Global Average Pooling Layer -

Global Average Pooling is an operation that perform average pooling of each channel of input feature map, means it's transform feature map from dimension HxWxK to 1x1xK by taking average of each channel (HxW) of feature map.

GAP Layer transforming feature map from 6x6x3 to 1x1x3 by taking average of each channel

Hence if you use GAP layer instead of flatten layer then it can handle any dimension of feature map and always produce 1x1xK dimension output where K is number of channel of input feature map which is always fixed for any CNN network. So next FC layer will always receive fixed dimension input.

As we are talking about GAP layer let's know other importance of this layer

• It is used in most of the network to handle image of different dimension 
• It is also used as a replacement of FC layer means output of GAP layer is directly fed to softmax layer
• Reduces number of trainable parameter of network and hence act as a regularizer
• Less prone to overfitting than traditional fully connected layer

Let's see VGG-16 network architecture with Global Average Pooling layer.

Here we can see in both the case output after global average pooling layer is 512 dimension vector.

Hence problem solved with GAP layer we can input smaller or even bigger image to a network if it have GAP layer.

Does this mean we can pass any dimension of image to this network ?? NO ! Why ??

You can see if a model is created for 224x224 image size then at the end before GAP layer dimension is reduced by 32 times ( 224/32 = 7 or 150/32 = 4), so our minimum dimension of input image should be greater than or equal to 32x32 image for VGG-16. 

Does this mean we can train VGG-16 having GAP layer with any image dimension (>= 32x32) ?? NO ! Why??

Now with GAP layer in VGG-16 network we can do inference with different dimension of images but not training, not directly at least, because at training time we train network in batches, batch of 32, 64 or 128 images, that means we pass multiple images to the network at the same time and if a batch contain different dimension of images then it will create a problem. Batch processing won't be able to handle different dimensions for different images. Solution for this is to create a image loader that load images of same dimensions in each batch.  

At inference time we always pass one image for inference so for inference we don't have any problem of different image dimension to network.

Notice one more thing number of parameters of network without GAP layer and with GAP layer, total parameter decreased from 138,357,544 to 37,694,248 so this proves the point that it act as a reguralizer and with GAP layer network is less prone to overfitting.

Code -

Let's see the code to create a model for inferencing different dimension input image. 

Here we are loading only convolutional block of VGG16 network not FC layer  and not passing any input dimension  for image, if you load full network with FC layer then you have to pass input dimension.

Let's say you want to train the model for 150x150x3 then at the time of data loading you have to resize the image to 150x150x3. Your model will be very accurate for this dimension but also be able to handle other dimensions images.

We can load imagenet weights, this can be helpful for fine-tuning.

We are adding Global Average Pooling Layer to network and adding two dense layer and output layer exactly same as in original VGG-16 network.

Now model is ready you can change number of classes and train on your dataset. Assuming model is trained let's see inference with this model.

Here I have passed one image without resizing and model is able to do inference on it. Original image size is 374x500x3

Let's reduce the image size.

You can see i have reduced the image dimension to 32x32x3 and still model is able to do inference.

Inference -

In my previous blog i talk about image classification and general model fine-tuning, I'm gonna use the same dataset & network and replace flatten layer to GAP layer and see how model is performing for different dimensions images.

Here are some prediction for you.
Loaded the VGG-16 network trained with GAP Layer, training images were resize to 224x224 at training time.
You can see below model is performing good for 32x32 dimension images also.

Now I passed the same image without resizing and image dimension is 499x403.

So we see here model trained for 224x224 images able to perform for different dimensions and quit accurately.

I won't say this is very great job as in real scenario it's very difficult to do correct prediction for such small images when model is trained on large dimensions images because in real scenario we get lot's of noisy data but this is quite GOOD.

Complete training and inference code is on GitHub. 

Conclusion -

• Because of flatten and fully connected layer, a CNN classification network can't process images of different dimensions.
• With global average pooling layer in any network we can do inference with different dimension images.
• Global average pooling layer reduce number of trainable parameter in network hence act as a regularizer and make network less prone to overfitting.

That's all for this blog, hope you find this blog informative.
Thanks for reading !!


Code and Model Link-

Code Link - https://github.com/111surajmaurya/VGG-16

Model File Link - https://drive.google.com/drive/folders/1H6Kgk2rIIfeZn31ecbkksRlpStqlKn00?usp=sharing