Batch Normalization: Accelerating Deep Network Training

In this blog we will talk about Batch Normalization layer.

Before understanding Batch Normalization first we need to understand what is internal covariant shift. 

Internal Covariant Shift is defined as the change in the distribution of  the network activation (feature map) due to change in the network parameter during training. 

So because of this internal covariant shift, training deep neural network is difficult because hidden layers input distribution keep changing, so the layers need to be continuously adapt to the new distribution. 

Theta2 need to readjust to re-compensate for the change in the distribution of x. This slow down the training by requiring lower learning rate and careful parameter initialization.  

To address this problem we can normalize the layer inputs, that's where batch normalization is useful.

Why we call this a Batch normalization because, training happens over a batch (64/128/256) of training images so we normalize batch of feature maps.

For a layer with k-dimensional input x = ( x(1).......x(k)), we will normalize each dimension, where the expectation and variance are computer over the training data set. Such normalization speed up the convergence.

The resulting normalized activation x^(k) have zero mean and unit variance.

Note that simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the non linearity.

To address this, we make sure that the transformation inserted in the network can represent the identity transform. To accomplish this we introduce, for each activation x(k), a pair of parameters γ(k), β(k), which scale and shift the normalized value. 

These parameters are learned along with the original model parameters, and restore the representation power of the network.  

Indeed, by setting γ(k)=√Var[x(k)] and β(k)=E[x(k)], we could recover the original activations, if that were the optimal thing to do. 

Batch Normalization During Training

• During Training, mean and variance of the each sample of mini-batch is calculated and for each sample mean and variance of each dimension (channels) are calculated separately. 
  
  

   

   
• Then we normalize the input feature map using calculate mean and variance to make the distribution with zero mean and unit variance.
  
  
   
   
• As we are changing (normalizing) the input of layer, this may change what layer can represent. To address this, we make sure that the transformation inserted in the network can represent the identity transform by using γ (scale) and β (shift) parameters.
  
  
  
  

Batch Norm during training - 

Batch Normalization During Inference

• The normalization of activations that depends on the mini-batch allows efficient training, but is neither necessary nor desirable during inference, we want the output to depend only on the input, deterministically.

• Instead of using mini-batch statistic we use population statistics that is moving average mean and variance of whole mini batch used in training. Moving mean and moving variance are non trainable parameters that are  calculated (updated) and stored during training.

  • moving_mean(t) = moving_mean(t-1)* momentum + mean(batch)(t) * (1 - momentum)
  • moving_var(t) = moving_var(t-1) * momentum + var(batch)(t) * (1 - momentum)
  • momentum = 0.99


• Then normalize the testing sample using population mean and variance. After that we need to transform this normalized sample using learned scale (γ) and shift (β) parameter of the respective layer to yield single linear transformation.

• Since the parameters are fixed in this transformation, the batch normalization procedure is essentially applying a linear transform to the activation.
  

One more important question for batch normalization is after which layer batch normalization should be used before activation layer or after activation layer.

• We can use batch normalization layer before or after activation layer
  

• For Sigmoid and Hyperbolic tangent (Tanh), s-shaped function, one can use batch normalization after activation function

• For activation which may result in non-Gaussian distribution like rectified linear activation function, one can use batch normalization before activation function

• Batch Normalization Paper suggests, Batch Normalization to use  before Activation Function
  

Let's see the output of batch normalization layers (Keras implementation), after a convolution layer whose output feature map dimension is (batch_size, 14, 14, 128). 

Observe the dimension of γ, β, moving mean and moving variance is equal to number of channel of input feature map.

That's all about Batch Normalization.

Thanks for reading the blog.

Reference

• https://arxiv.org/pdf/1502.03167.pdf
• https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/
  

Dropout

What is Dropout Layer ?

Dropout layer randomly sets input units to 0 with a frequency of drop_rate at each iteration during training time, which help prevent overfitting.  

The key idea of dropout is randomly drop nodes along with their connection from the neural network during training time.

Dropout layer takes single float values as input between 0 to 1. In Keras implementation it denotes drop probability of unit. We will call it p_drop, so keep probability of unit is p_keep = 1 - p_drop.

Each unit is retained with a fixed probability (p_keep) independent of the other units. Generally we use p_drop 0.5 for dense layer.

Why do we need Dropout ? 

To solve the problem of Overfitting.

Overfitting means our model is performing well on training data but not performing well on test data (or new data).

One of the reason for overfitting is because our model is quite complex (having large number of parameter), so instead of just learning (generalizing) patterns/features in the data it also learn the noise present in the data and so it adjust it's weight to perform well on training data or we can also say that it adjust it's weight to memories the training data. And other reason of overfitting is training data is not good representation of overall (real) data. 

If training dataset is good representation of real data but not in good amount, this can also cause overfitting.

How dropout is solving the problem of Overfitting.

Multiple way to look into this

1. One way to look into this is, it reduces model complexity by randomly setting layer units to zero and so reducing model complexity that help in solving the overfitting.

2. During each training step, it drops unit with p_drop probability from the layer and then train a thinned network. 

Because at each training step, it trains a unique thinned network with less neurons, so the neuron present in network learn the representation(features) required for correct prediction. This prevent neurons from co-adapting too much on each other.

This make the network capable of better generalization and hence solving overfitting.

3. ""Overfitting can also be solved by training all possible neural network for a dataset and average the prediction form all model. But this is not possible.""

Let's see how we can interpret the above concept with dropout layer.

During training with dropout we train multiple sparse (thinned) neural network and at test time, we approximate the effect of averaging the predictions from all these thinned network by simply using original unthinned network that has smaller weights. This help in solving overfitting problem. Let's see in detail.

A neural network with n units, can be seen as 2^n possible thinned neural network and all these network shares the same weights.

During each training step we sample one out of 2^n network and train, so during whole training process we train multiple thinned network.

So training a neural network with dropout can be seen as training a collection of 2^n thinned network with extensive weight sharing, where each thinned network get trained very rarely, if at all. 

At test time, we can not take average of the prediction from all those networks. However simple approximate average method work well. So during inference time, idea is to use full network with all units with scaled-down version of weights. 

If a unit is retained with p_keep during training, then outgoing weights of that unit are multiplied by p_keep at test time. This ensure the expected out of hidden unit is same as the actual output at test time. By doing this scaling, 2^n network with shared weights can be combined into a single neural network to be used at test time.

 

Dropout during training and inference time

Lets say we want to apply dropout on this input data d = {1,2,3,4,5} with p_drop = 0.2 so now during training any one unit of d will become zero and d could be {1, 2, 3, 0, 5} because p_drop is 0.2 another way to look into this is we keep each node with probability (p_keep) 0.8 .

During inference time we will be using all the unit as dropout don't remove units during inference time. If we use all unit during inference, expected output will be different than training time. To make sure that the distribution of the values after the transformation during inference time remains almost the same, we multiply input with keep probability p_keep(1-p_drop) at inference time, during inference same d would be set to {0.8, 1.6, 2.4, 3.2, 4.0}. 

But in general we don't want to do anything with dropout layer during inference time so during training time only, we scale the values by 1/p_keep.

So now during training d could be set to {1.25, 2.5, 3.75, 0, 6.25} and nothing will happen with the input d during inference time.

That is why if you see Keras documentation of dropout it will say, dropout first set units to 0 with given drop probability(p) and then scale the remaining values by 1/(1-p).

That's all about dropout, thanks for reading the blog!

References

1. https://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf

2. https://keras.io/api/layers/regularization_layers/dropout/

3. https://leimao.github.io/blog/Dropout-Explained/

CNN vs NN : FASHION MNIST

In this blog we will compare Neural Network (NN) and Convolutional Neural Network (CNN) on Fashion MNIST dataset. Basically comparison of NN and CNN based on model convergence, model accuracy after N epoch and inference on translated images. We will also verify CNN translation invariant property.

You can read more about classification network training, transfer learning and inference in my previous blogs Image Classification in Keras and VGG-16 Inference with different image dimension.

Let's get started with this blog.

Dataset

Fashion MNIST dataset have 60,000 training images and 10,000 testing images. Each image is a 28x28 grayscale image, associated with a label from 10 classes.

One can check this google colab Notebook to follow this blog.

Import

First let's import required libraries.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import cv2
 

Load Dataset

Load Fashion MNIST dataset directly from tensorflow default datasets.

mnist = tf.keras.datasets.fashion_mnist
(training_images,training_labels),(test_images,test_labels)=mnist.load_data()

These are the 10 classes of Fashion MNIST.

classes =['T-shirt/top', 'Trouser', 'Pullover', 'Dress',
'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot' ]

After loading training and testing dataset, let's visualize few training images.

fig=plt.figure(figsize=(8, 8))
row = 3; col = 4
for i in range(row*col):
  fig.add_subplot(row, col, i+1).set_title(
    str(classes[training_labels[i]]))
  plt.imshow(training_images[i])
plt.show()

Fig. Sample training images (28x28x1)

Data Preprocessing

For model training we always want datasets to be normalized in -1 to 1 or 0 to 1 range. As normalize data help in model convergence. So let's normalize images by dividing them by 255.

training_images, test_images = training_images/255.0, test_images/255.0

Network Definition

Now we will define NN and CNN network. In NN we are having two hidden Dense layer with 256 units each and one output Dense layer with 10 units for 10 classes.

def get_NN_model():
  model = tf.keras.models.Sequential()
  model.add(tf.keras.layers.Flatten(input_shape=(28,28)))
  model.add(tf.keras.layers.Dense(256, activation= 'relu'))
  model.add(tf.keras.layers.Dense(256, activation= 'relu'))
  model.add(tf.keras.layers.Dense(10, activation= 'softmax'))
  model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
  return model
  

In CNN network we are have three Conv2D layer with 64, 128 and 128 kernel respectively followed by MaxPooling layer and one dense layer for output.

def get_CNN_model():
  model = tf.keras.models.Sequential()
  model.add(tf.keras.layers.Conv2D(64,(3,3), input_shape=(28,28,1), padding = 'same', activation = 'relu'))
  model.add(tf.keras.layers.MaxPooling2D((2,2)))
  model.add(tf.keras.layers.Conv2D(128,(3,3), padding = 'same', activation = 'relu'))
  model.add(tf.keras.layers.MaxPooling2D((2,2)))
  model.add(tf.keras.layers.Conv2D(128,(3,3), padding = 'same', activation = 'relu'))
  model.add(tf.keras.layers.GlobalAveragePooling2D())
  # model.add(tf.keras.layers.Flatten())
  # model.add(tf.keras.layers.Dense(256, activation = 'relu'))
  model.add(tf.keras.layers.Dense(10, activation = 'softmax'))
  model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
  return model

We will train both network for 30 epochs and then compare their accuracies, if any network reaches 99% accuracy before 30 epochs then we will stop training using below callback function.

class epoch_callback(tf.keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs={}):
    if(logs.get('accuracy') >= 0.99):
      self.model.stop_training = True

Let's create both network and print their summary.

nn_model = get_NN_model()
print(nn_model.summary())
cnn_model = get_CNN_model()
print(cnn_model.summary())

Model Summary

Note the number of learning parameters for both the network are in same range, not much difference and using GlobalAveragePooling layer in CNN network has reduced significant number of learning parameter.

#NN model summary
Layer (type)                 Output Shape              Param #   
=================================================================
flatten (Flatten)            (None, 784)               0         
_________________________________________________________________
dense (Dense)                (None, 256)               200960    
_________________________________________________________________
dense_1 (Dense)              (None, 256)               65792     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                2570      
=================================================================
Total params: 269,322
Trainable params: 269,322
Non-trainable params: 0

#CNN model summary
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 28, 28, 64)        640       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 64)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 14, 14, 128)       73856     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 128)         0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 7, 7, 128)         147584    
_________________________________________________________________
global_average_pooling2d (Gl (None, 128)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 10)                1290      
=================================================================
Total params: 223,370
Trainable params: 223,370
Non-trainable params: 0

Training NN & CNN

Let's train both network for 30 epochs using tensorflow fit function.

filepath = '/content/nn_model.h5'
callback = epoch_callback()

checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath,
    monitor='val_loss', save_best_only=True, mode='auto')

history_nn = nn_model.fit(
    training_images, 
    training_labels, 
    validation_data=(test_images, test_labels), 
    epochs=30, 
    callbacks=[checkpoint,callback])

Above code will train the NN with around 95% training and 88% validation accuracy.

To train the CNN we have to increase the image dimension to add image channel information as Conv2D layer of above CNN network takes 28x28x1 dimension images.

training_images_cnn = np.expand_dims(training_images, axis=3)
test_images_cnn = np.expand_dims(test_images, axis=3)
print(training_images_cnn.shape, test_images_cnn.shape)

Next, train CNN network for 30 epochs.

filepath = '/content/cnn_model.h5'
callback = epoch_callback()

checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath,
    monitor='val_loss', save_best_only=True, mode='auto')

history_cnn = cnn_model.fit(
    training_images_cnn, 
    training_labels, 
    validation_data=(test_images_cnn, test_labels),
    epochs=30, 
    callbacks = [checkpoint,callback])

Plot training history for both the network.

def plot_model_history(history, title):
  plt.plot(history.history['accuracy'])
  plt.plot(history.history['val_accuracy'])
  plt.title(title)
  plt.ylabel('accuracy')
  plt.xlabel('epoch')
  plt.legend(['train', 'val'], loc='upper left')
  plt.show()

plot_model_history(history_nn, 'NN')
plot_model_history(history_cnn, 'CNN')

From below plot we can see training accuracy for both network keep increasing but validation accuracy for NN network saturate after 88% where as CNN achieve 92% accuracy for same number of epoch. 

NN epoch vs accuracy

CNN epoch vs accuracy

Model Evaluation

Evaluate best model of both the network on test data to verify accuracy and loss.

nn_model.load_weights('/content/nn_model.h5')
cnn_model.load_weights('/content/cnn_model.h5')

evaluation_nn = nn_model.evaluate(test_images, test_labels)
evaluation_cnn = cnn_model.evaluate(test_images_cnn, test_labels)

print('NN loss : ', evaluation_nn[0], ' accuracy: ', evaluation_nn[1], 'on test data.')
print('CNN loss : ', evaluation_cnn[0], ' accuracy: ', evaluation_cnn[1], 'on test data.')

313/313 [==============================] - 1s 2ms/step - loss: 0.3344 - accuracy: 0.8767
313/313 [==============================] - 1s 2ms/step - loss: 0.2258 - accuracy: 0.9212
NN loss :  0.3343818485736847  accuracy:  0.8766999840736389 on test data.
CNN loss :  0.22580283880233765  accuracy:  0.9211999773979187 on test data.

Till now we have seen that CNN model converges faster than NN and achieve better training and validation accuracy for same number of epochs.

Inference

Now let's see how CNN and NN performs on test images and on slightly translated and bigger images. We will create a new image 38x38 and copy original image on top corner or bottom corner of new image and then resizing it to 28x28 for inference.

Let's define a common function for prediction, it takes model and image and return predicted class and probability for image.

def predict(model, img_nn):
  pred = model.predict(img_nn)
  cls_id = np.argmax(pred)
  conf = pred[0][cls_id]
  cls_name = classes[cls_id]
  return cls_name, conf

Below is the preprocessing function for original image. It return different dimensions image for NN and CNN, as NN needs 28x28 (or 28x28x1) and CNN needs 1x28x28x1 dimension images at inference time.

def preprocess_img(img):
  img_nn = np.expand_dims(img, axis=0) 
  img_cnn = np.expand_dims(img_nn, axis=3)
  return img_nn, img_cnn

Define a function for creating new image for NN and CNN, which will be the translated form of test images.

def translate_object(img):
  new_img = np.zeros((38,38))
  if(np.random.randint(2)):
    new_img[10:38,10:38] = img
  else:
    new_img[0:28,0:28] = img
  
  new_img = cv2.resize(new_img, (28,28))
  plt.imshow(new_img)
  plt.show()
  new_img =np.expand_dims(new_img, axis=0)
  img_nn = np.copy(new_img)
  img_cnn =np.expand_dims(new_img, axis=3)
  return img_nn, img_cnn

Finally let's process few images and it's translated version from the test dataset and see prediction from NN & CNN network.

for i in range(len(test_images)):
  
  img = test_images[i]
  img_nn, img_cnn = preprocess_img(img)
  plt.imshow(img)
  plt.show()
  print('GT-class: ', classes[test_labels[i]])
  print('NN on org Image ', predict(nn_model, img_nn))
  print('CNN on org Image ', predict(cnn_model, img_cnn))
  
  img_nn, img_cnn = translate_object(img)

  predict(nn_model, img_nn)
  predict(cnn_model, img_cnn)
  print('NN on transform Image ', predict(nn_model, img_nn))
  print('CNN on transform Image ', predict(cnn_model, img_cnn))

  print('\n**************************************************************\n')
  if(i == 10):
    break
  #break 

Result 

For above image we can see NN and CNN predicted correct class but when same image underwent some spatial translation neural network predict wrong class whereas CNN predict correctly, however we can see drop in confidence value for the class from 0.99 to 0.42.

Conclusion 

• Convolutional neural network converges faster than Neural network and achieve better accuracy on test dataset.
• CNN perform better than NN on translated form of original image. 

That's all for this blog. Code is available on Github for experiment.

Thanks!!

VGG-16 Inference with different image dimension

In this blog i will talk about how to create a classification network or fine-tune any pre-trained classification network (VGG-16) that accepts image of any dimension rather than one dimension on which it is trained. Generally a model trained for MxNx3 dimension images accept only MxNx3 input not other dimensions but here we will see how to modify network to accept different dimension input image also.

I will explain this with VGG-16 network.

(**Dimension is referred to only width and height of image/feature map not channel/depth) 

Let's talk about scenario where a single network accepting different dimensions input can be helpful.

• Case -1 You have few hundred images of smaller dimension (let's say 100x100x3) then fine-tuning a pre-trained network is one of the best option you have instead of training one from scratch.
• Case-2 You trained a classification network for a fixed input dimension (224x224x3) and at inference time you get different dimension input ( ranging from 200x200x3 to 300x300x3) and you don't want to pad or resize inference images to loose information.
• Case-3 You have multiple dataset with different input dimension then with this kind of network you can easily train a classifier without modifying input dataset.

There can be other scenario where it is helpful but let's not talk about all and move forward with solution.

So the first question is what's the problem if we pass different dimensions images to a VGG-16 network, which layer will create problem convolution or pooling or flatten or fully connect (FC) layer ??

Let's see what these layers do in brief.

• Convolution layer accept any dimension input and perform convolution with kernel values and its output dimensions depend on the padding or stride of kernel. It may or may not reduce dimension.
• Pooling layer accept any dimension input and its output dimension depend on stride of pooling operation. It will reduce dimension.
• Flatten layer accept any dimension input and its output is reshaped input in single dimension.
• Fully Connect (FC) layer accept fixed input dimension and its output dimension depend on next FC layer input dimension or output layer dimension i.e. both are fixed dimensions. It may or may not reduce dimension.

Now let's see network architecture for original VGG-16 which is trained on 224x224x3 images and same VGG-16 network when trained on 150x150x3 images.

For different dimensions of input images, after block5_pool layer feature map (feature map is nothing but the output of convolution or pooling layer of CNN ) dimensions is different because of convolution and pooling layer as they reduce feature map size by some constant factor and after that flatten layer is just flattening the feature map to one dimensional vector form.

We can see input to first FC layer is 25088 (7x7x512) when image is 224x224x3 and 8192 (4x4x512) when input is 150x150x3, so this will create a problem if you pass 150x150x3 image to a network which is trained for 224x224x3

In the above image you can see i have loaded original VGG-16 model for 1000 classes, and it gave error for 374x500x3 dimension input image, but if you uncomment resize line then it will run and give probability for 1000 classes.

So a network trained for 224x224x3 will take only 224x224x3 dimension input not 150x150x3 and vice versa. 

If we want a single network to accept both images then output dimension of flatten layer should be fixed so that FC layer should always accept the output of flatten layer. The problem will be solved if somehow we always pass fixed input dimension to FC layer And that's where Global Average Pooling Layer help us.

Conclusion till here is because FC layer accept fixed length input that's why passing different dimension image to VGG-16 network results in the error. 

Global Average Pooling Layer -

Global Average Pooling is an operation that perform average pooling of each channel of input feature map, means it's transform feature map from dimension HxWxK to 1x1xK by taking average of each channel (HxW) of feature map.

GAP Layer transforming feature map from 6x6x3 to 1x1x3 by taking average of each channel

Hence if you use GAP layer instead of flatten layer then it can handle any dimension of feature map and always produce 1x1xK dimension output where K is number of channel of input feature map which is always fixed for any CNN network. So next FC layer will always receive fixed dimension input.

As we are talking about GAP layer let's know other importance of this layer

• It is used in most of the network to handle image of different dimension 
• It is also used as a replacement of FC layer means output of GAP layer is directly fed to softmax layer
• Reduces number of trainable parameter of network and hence act as a regularizer
• Less prone to overfitting than traditional fully connected layer

Let's see VGG-16 network architecture with Global Average Pooling layer.

Here we can see in both the case output after global average pooling layer is 512 dimension vector.

Hence problem solved with GAP layer we can input smaller or even bigger image to a network if it have GAP layer.

Does this mean we can pass any dimension of image to this network ?? NO ! Why ??

You can see if a model is created for 224x224 image size then at the end before GAP layer dimension is reduced by 32 times ( 224/32 = 7 or 150/32 = 4), so our minimum dimension of input image should be greater than or equal to 32x32 image for VGG-16. 

Does this mean we can train VGG-16 having GAP layer with any image dimension (>= 32x32) ?? NO ! Why??

Now with GAP layer in VGG-16 network we can do inference with different dimension of images but not training, not directly at least, because at training time we train network in batches, batch of 32, 64 or 128 images, that means we pass multiple images to the network at the same time and if a batch contain different dimension of images then it will create a problem. Batch processing won't be able to handle different dimensions for different images. Solution for this is to create a image loader that load images of same dimensions in each batch.  

At inference time we always pass one image for inference so for inference we don't have any problem of different image dimension to network.

Notice one more thing number of parameters of network without GAP layer and with GAP layer, total parameter decreased from 138,357,544 to 37,694,248 so this proves the point that it act as a reguralizer and with GAP layer network is less prone to overfitting.

Code -

Let's see the code to create a model for inferencing different dimension input image. 

Here we are loading only convolutional block of VGG16 network not FC layer  and not passing any input dimension  for image, if you load full network with FC layer then you have to pass input dimension.

Let's say you want to train the model for 150x150x3 then at the time of data loading you have to resize the image to 150x150x3. Your model will be very accurate for this dimension but also be able to handle other dimensions images.

We can load imagenet weights, this can be helpful for fine-tuning.

We are adding Global Average Pooling Layer to network and adding two dense layer and output layer exactly same as in original VGG-16 network.

Now model is ready you can change number of classes and train on your dataset. Assuming model is trained let's see inference with this model.

Here I have passed one image without resizing and model is able to do inference on it. Original image size is 374x500x3

Let's reduce the image size.

You can see i have reduced the image dimension to 32x32x3 and still model is able to do inference.

Inference -

In my previous blog i talk about image classification and general model fine-tuning, I'm gonna use the same dataset & network and replace flatten layer to GAP layer and see how model is performing for different dimensions images.

Here are some prediction for you.
Loaded the VGG-16 network trained with GAP Layer, training images were resize to 224x224 at training time.
You can see below model is performing good for 32x32 dimension images also.

Now I passed the same image without resizing and image dimension is 499x403.

So we see here model trained for 224x224 images able to perform for different dimensions and quit accurately.

I won't say this is very great job as in real scenario it's very difficult to do correct prediction for such small images when model is trained on large dimensions images because in real scenario we get lot's of noisy data but this is quite GOOD.

Complete training and inference code is on GitHub. 

Conclusion -

• Because of flatten and fully connected layer, a CNN classification network can't process images of different dimensions.
• With global average pooling layer in any network we can do inference with different dimension images.
• Global average pooling layer reduce number of trainable parameter in network hence act as a regularizer and make network less prone to overfitting.

That's all for this blog, hope you find this blog informative.
Thanks for reading !!


Code and Model Link-

Code Link - https://github.com/111surajmaurya/VGG-16

Model File Link - https://drive.google.com/drive/folders/1H6Kgk2rIIfeZn31ecbkksRlpStqlKn00?usp=sharing

Image Classification in Keras

In this blog I will explain how to do image classification in python using Keras. Image classification is a basic problem in Deep Learning. It is a method to classify images into their respective classes.

image_1.jpg

image_2.jpg

Here is an example of Image classification, image_1.jpg should be predicted as cat and image_2.jpg should be predicted as dog by CNN (Convolutional Neural Network) model.

Keras gives us lots of CNN (VGG, ResNet, Inception etc.) models already trained on ImageNet Dataset for 1000 classes (http://www.image-net.org/), we will be using VGG-16 model and fine-tune it on dog-vs-cat dataset (https://www.kaggle.com/c/dogs-vs-cats/data) for two classes.

Data Preparation 

How to prepare Train, Validation and Test dataset ?

Train dataset is the only dataset that is used for model training (learning weights and biases) and validation dataset is used to monitor the loss and accuracy of intermediate model on unseen data. Train and validation dataset is used during training only.

After training we can choose top three model which is performing good on validation data, and test these models on the test dataset to select the best model.

We will use 80% (8000) of the total data (10000) for training purpose and 10%-10% (1000-1000) for validation and testing. It’s not a rule to use 10% of the whole dataset for testing and validation, sometimes we also use 1-2% of total images when we have millions of images.

We need three directory train, validation & test. Each directory should contain one sub-directory for each class filled with respective images.

-- train/

     -- cat/

     --dog/

--validation/

     --cat/

     --dog/

--test/

     --cat/

     --dog/

Code and Explanation

In this section i will explain code and important points regarding classification model.

First let's import all the required packages and modules.

As VGG-16 model is trained on 224x224x3 image size so we will be using same size for fine tuning however we can also fine-tune it for different size like 100x100x3 or 448x448x3, we will see how to fine-tune model for different image size in my future blogs.

As we want to modify the VGG-16 model for our own dataset we will load only convolution blocks not fully connected and output layers. This line will load the VGG-16 model with ImageNet weights without fully connected and output layers as we set “include_top” argument as false.

Original VGG-16 have two fully connected layers with 4096 neurons and output layer for 1000 classes.

As cat and dog images are not very difficult to differentiate so we will be using only 256 neurons in fully connected (FC) layer instead of 4096 neurons as in original VGG-16 network. Final prediction layer will have only one neuron for binary classification.

You can see summary of our network using model.summary() function.

Freeze Layer

Freezing any layer means we won't be modifying the weights and biases of that layer during training. This is one of the important things to do while fine tuning any already pre-trained model.

Why to freeze layers ?

As we know each layer in any network learns to detect some kind of feature like different kind of edges, corners, patches, color and template like feature from images so during fine tuning pre-trained model we can use some of the features (like edge and corner features) as it is useful for all problem statement. That's why we can freeze some layers to use already learned features.

If we freeze some layers of network that means we are modifying less layers during training process, so training and convergence process of model should be fast.

When to freeze the layers of the base model ??

One can freeze some layers during fine-tuning of pre-trained model in case of

• If you have less training images
• If new classes are similar to old classes on which model is already trained, and if new classes are totally different then one would have to train more layers of the base network like in case of fine-tuning VGG model on medical images.

Currently we are training only 30% layers of base model, If we have more images we can train more layers.

Note: Layer freezing of model should be done before model compilation.

Now let's compile the model with SGD optimizer and binary crossentropy loss. For more than two classes use categorical_crossentropy.

One can use Adam optimizer or other optimizer because some time some optimizer perform well on some data and the same optimizer with same parameters doesn’t perform well on different types of data. So it’s always better to experiment with two-three optimizers and select the one which performs well on your data.

Data Loading

Let's see how to load the data and perform augmentation using ImageDataGenerator class of Keras.

At training and validation step we want to normalize the data and at training time we also want to augment the images to increase the training dataset size. Here we are using shear, zoom and horizontal flip techniques to augment more training images.

flow_from_directory is the function that load the images from given directory, we have to pass the path of the directory containing sub-directory for each classes. Here we are using 'binary' class_mode because we have only two classes, for more than two classes use 'categorical' class_mode.

Callback

A callback is a set of functions to be applied at given stages of the training procedure. With the help of callback functions we can monitor the loss, save weights files and plot training & validation loss graphs during training.

To make things simple we will be using only ModelCheckpoint callback for now. With this callback we can save weights when their is increment in validation accuracy or decrement in validation loss controlled by monitor and mode parameter.

Enabling save_best_only parameter will save weight file if new weights are better than the previous saved weight, disabling it will save weight file of all epochs.

save_weights_only is the important parameter, if it’s true then it will save only weights of the network not the network architecture. To use this weights you have to first create the network then you can load the weights with load_weights function.

If save_weights_only is false then while saving weights it will save network architecture, training configuration and state of optimizer which allow you to resume the training where you left off. You don’t need network architecture information, you can directly load weights and network by using load_model function.

By default save_weights_only is False.

Training

Now we will use fit_generator function to start the training.

There are few terms that one should know.

Epoch - While training any deep learning model, the model is training on the whole dataset many times not just once. Epoch defined as one pass over the entire dataset. Model trained for one epoch means models have seen the whole training images once.

Batch Size - A set of N images. The samples in a batch are processed independently, in parallel. During training weights are going to be updated after each batch. Batch size can be 32, 64, 128 or 256 , depending on GPU and memory size of your system use appropriate batch size. Batch size also affects the convergence of the model I will not go in detail of this.

Steps per epoch - steps_per_epoch * batch_size = total training samples

So steps per epoch should be equal to total_training_sample divided by batch_size if you set less than the above calculated number then you will not use all training images during one epoch.

Let's train this network for 25 epoch.

Here is training summary till 8th epoch. Model achieved 98% accuracy in 6th epoch itself.

Inference
Let's see how to do inference on test images.

 Import libraries to load model and to read images.

Here we have defined the classes and loaded the model using load_model function.

In the above code we have loaded the image using PIL image processing library (I have used PIL library because Keras data loader internally uses PIL to read the images at training time), and resized the image to (224,224,3) as our model is trained for this size and then we have added one more dimension in image to make image dimension from (224,224,3) to (1,224,224,3). Model accept four dimension input, added dimension is to represent batch_size. And finally we have normalized the image by dividing it by 255.

Predict function on image for binary classification give us the probability between 0 & 1. Put a threshold on probability if less then threshold then class 0 else class 1 and here we have used 0.5 as threshold value. If model is biased toward one class then you might want to change the threshold value.

So in this blog we have seen how to fine tune VGG-16 model for new dataset & achieve good accuracy and how to use new trained model for inference on images.

That's all for this blog, hope you find this blog informative.
Thanks for reading !!

Code, used dataset  and model link are below.

Full code github link - https://github.com/111surajmaurya/VGG-16

Dataset google drive link - https://drive.google.com/drive/folders/15Et639VosTWxEmPLRqx4hdTdGvzQupbg?usp=sharing

Model link - https://drive.google.com/drive/folders/1F72YB-D4XOlIvnujE--HPj9lMUkXsjKe?usp=sharing