Post #261

#ML#self-supervised #representation Contrastive loss is widely used in representation learning. However, the mechanism behind it is not as straightforward as it seems. Wang & Isola proposed a method to rewrite the contrastive loss in to alignment and uniformity. Samples in the feature space are normalized to unit vectors. These vectors are allocated onto a hypersphere. The two components of the contrastive loss are - alignment, which forces the positive samples to be aligned on the hypersphere, and - uniformity, which distributes the samples uniformly on the hypersphere. By optimization of such objectives, the samples are distributed on a hypersphere, with similar samples clustered, i.e., pointing to the similar directions. Uniformity makes sure the samples are using the whole hypersphere so we don't waste "space". References: Wang T, Isola P. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2005.10242