Overview / Usage

In this report, we are going to discuss a new training methodology namely Supervised Contrastive Learning introduced in https://arxiv.org/abs/2004.11362 by Khosla et al. This methodology is an adaption of contrastive learning in the field of fully supervised learning problems. This method actually uses the label information to cluster the samples belonging to the same class in the embedding space. On top of it, a linear classifier can be used to classify the images. This method is said to be outperforming the Cross-Entropy.

Methodology / Approach

The idea proposed in Supervised Contrastive learning is pretty simple. Learn how to map the normalized encoding of samples belonging to the same category closer and the samples belonging to the other classes further embeddings of all cat images should be similar with each other and distant from across all dog images and opposite.

We know that the neural network first converts the image into a representation and then this representation is used to predict the result. So if the representations are formed keeping the idea given above into consideration, it will be easier for the classifier to give accurate results.

This whole idea is achieved in two stages.

Stage 1: In this stage, the network is trained using contrastive loss. Here the images are encoded in such a way that embeddings of one class are close and that of other classes are far. And to do this, the label of the images are used. This stage has three components namely data augmentation module, encoder network, and encoder network. These components are explained below separately.

Stage 2: Here, the encoder network used in Stage 1 is frozen and the projector network is discarded. The representation learned from the encoder network is then used to learn a classifier which is nothing but a linear layer. At this stage, Cross-Entropy loss is used to predict the labels.

Let's have a look at the components of Stage 1 of the training.

​

Full report is available here.

Technologies Used

TensorFlow 2

Repository

https://github.com/sayakpaul/Supervised-Contrastive-Learning-in-TensorFlow-2