Abstract: Recent self-supervised video representation learning methods have found significant success by exploring essential properties of videos, e.g. speed, temporal order, etc.

This work exploits an essential yet under-explored property of videos, the \textit{video continuity}, to obtain supervision signals in self-supervised representation learning.

Specifically, we formulate three novel continuity-related pretext tasks, i.e. continuity justification, discontinuity localization, and missing section approximation, that supervise a shared backbone for video representation learning.

This self-supervision approach, termed as Continuity Perception Network (CPNet), encourages its backbone network to learn local and long-ranged motion and context representations and outperforms prior arts on multiple downstream tasks, such as action recognition, video retrieval, and action localization.

Also, the video continuity can be complementary to other video properties for representation learning, and integrating the proposed pretext task to prior arts can yield much performance gains.