多任务对比学习的自监督视频表达

Multitask Contrastive Learning for Self-Supervised Video Representation

摘要: 现有的自监督学习使用单一的空间或时间代理任务。单一的代理任务，从未标记的数据中提供单一的监督信号，不足以描述视频表示学习的空间特征和时间特征之间的差异。在本文中，我们提出了一个多任务对比学习方法，它通过对多个时空代理任务的对比学习，在时空自注意力的情况下学习有区别的时空特征。不同的空间代理任务学习不同的空间特征，包括空间旋转和空间拼图。不同的时间代理任务学习不同的时间特征，包括时间顺序和时间节奏。我们将视频表示为每个代理任务的多个不同特征，并设计基于代理任务的对比损失来分离一个视频中学习的空间特征和时间特征。基于代理任务的对比损失鼓励不同代理任务学习不同的特征，同一代理任务学习相似的特征，可以学习到同一视频中每个代理任务的判别特征。实验表明，在UCF-101数据集和HMDB-51数据集的行为识别上优于现有的自监督学习方法。

Abstract: Most existing self-supervised works use a single spatial or temporal pretext task. A single pretext task, providing single supervision from unlabeled data, is insufficient to describe the difference between spatial features and temporal features for video representation learning. In this paper, we propose an attentive spatiotemporal contrastive learning network, which learns discriminative spatial-temporal features with self-attention by contrastive learning between multiple spatial and temporal pretext tasks. Different spatial features are learned by multiple spatial pretext tasks, including spatial rotation, and spatial jigsaw. Different temporal features are learned by multiple temporal pretext tasks, including temporal order, and temporal pace. We represent video as multiple different features for each pretext task, and design pretext task-based contrastive loss to separate the spatial feature and the temporal feature learned in one video. The pretext task-based contrastive loss encourages the different pretext tasks to learn dissimilar features and the same pretext task to learn similar features, which can learn the discriminative features for each pretext task in one video. Experiments show that it outperforms existing self-supervised learning methods for behavior recognition on the UCF-101 dataset and the HMDB-51 dataset.