基于不确定性感知自适应伪标签的指代视频目标分割

Uncertainty-Aware Adaptive Pseudo-Labeling for Referring Video Object Segmentation

摘要: 指代视频目标分割(Referring Video Object Segmentation, RVOS)是一项新兴的多模态任务，旨在通过理解给定指代表达的语义来分割视频片段中的目标区域。然而，基准数据集的标注是通过半监督方式收集的，仅提供了视频第一帧的真实目标掩码。为了在一个更综合的框架中探索未标记数据中的隐藏知识，本文引入了在线伪标签来解决RVOS问题。具体来说，使用之前训练阶段的即时学习检查点作为教师模型，在未标记的视频帧上生成伪标签，并将获得的伪标签用作训练数据的增强，以监督随后的训练阶段。为了避免伪标签带来的混淆，本文提出了一种不确定性感知的细化策略，根据模型预测的置信度自适应地修正生成的伪标签。本文在基准数据集Refer-YouTube-VOS和Refer-DAVIS₁₇上进行了广泛的实验来验证所提出的方法。实验结果表明，本文的模型与最先进的模型相比取得了具有竞争力的结果。

Abstract: Referring video object segmentation (RVOS) is an emerging multimodal task aiming to segment target regions in video clips by understanding the semantics of given referring expressions. While the annotations of the benchmark datasets are collected in a semi-supervised manner, which only provides the ground truth object masks on the first frame of videos. To explore the concealed knowledge in the unlabeled data in a more integrated framework, we introduce online pseudo-labeling to address RVOS. Specifically, we employ the on-the-fly learned checkpoints in the previous training epochs as the teacher model to produce the pseudo labels on the unlabeled video frames, and the obtained pseudo-labels are utilized as augmentation for the training data to supervise the subsequent training stage. To avert the confusion derived from pseudo-labels, we propose an uncertainty-aware refinement strategy to adaptively rectify the generated pseudo-labels based on the model prediction confidence. We conduct extensive experiments on the benchmark datasets Refer-YouTube-VOS and Refer-DAVIS₁₇ to validate the proposed approach. The experimental results demonstrate that our model achieves competitive results compared with state-of-the-art models.