Overview

SSL has allowed pretraining of neural networks to scale beyond the size of labelled datasets, demonstrating robust performance without the need for costly annotation. This approach has successfully scaled training dataset sizes to billions of images.

However, state-of-the-art (SoTA) models often learn representations limited to single-image inputs, lacking temporal context. Visual representations derived from such static images are limited to learning from disjoint static snapshots of the world. This limitation is particularly pronounced in recent SSL techniques, all of which are trained on meticulously curated and object-centric datasets, such as ImageNet. Attempts to scale up single-image techniques to larger, less-curated datasets like Instagram-1B have not yielded substantial improvements in performance. A single image, regardless of artificial augmentation, has its constraints: it cannot create new perspectives of an object or anticipate unfolding events in a scene.

The primary goal of this tutorial is to introduce to the computer vision community, the concept of learning robust representations by leveraging the rich information in video frames. While, image-based pretraining has gained recent popularity with SimCLR, the practice of pretraining models from videos dates back much earlier. This tutorial will recapitulate both early and recent works, which have pretrained image encoders using videos for different pretext tasks such as egomotion prediction, active recognition, dense prediction etc. We also discuss practical implementation details relevant for practitioners and highlight connections to other, existing works such as VITO, TimeTuning, DoRA, V-JEPA etc. We also discuss recent works aimed to mimic human visual systems such learning from one continuous video stream and by learning from longitudinal audio-visual headcam recordings from young children, thereby putting this concept into a broader context.

Motivation

These works from the last 6-12 months demonstrate a paradigm shift, showcasing SSL models pretrained on videos can outperform image-based pretraining. The notable progress validates this trend, making this tutorial on past and present advancements crucial for newcomers in the field. Key questions we aim to tackle in this tutorial include:

• Can we learn strong image encoders from good quality videos (i.e. with limited data)?
• Do we need synthetic augmentations? How useful are the natural augmentations in videos?
• Can we learn from a continuous stream similar to humans?