基于自监督学习方法的半监督视频目标分割研究

A Study on Self-supervised Learning for Semi-supervised Video Object Segmentation

本文主要研究自监督学习方法在视频目标分割中的应用。首先通过挖掘大规模无标注视频数据中的时间-空间关系,让神经网络作为特征编码器学习视频帧之间的相似性和连续性;然后通过记忆力机制训练网络,使其对当前帧和多个参考帧之间的关系进行建模;利用特征编码器学习到的特征对视频帧中的分割目标进行重建,进行下游的视频目标分割任务;最后,利用在线自适应模块对视频分割结果的错误进行修正。实验结果表明,本文的自监督方法在视频分割任务上的表现可以更加接近有监督方法的分割结果,采用记忆力机制和在线自适应模块可以大大提高视频目标分割的准确性。另外本文探究了数据有效性,当采用少量数据进行网络的自监督训练时,模型仍能取得较好的效果,意味着这个任务本身不需要大规模数据集中富含的复杂语义信息进行建模。

This paper mainly studies self-supervised representation learning for semi-supervised Video Object Segmentation(semi-VOS). Firstly, we leverage the spatio-temporal coherence in videos to enable the model to learn correspondence among video frames. Then, we propose a momentum memory mechanism to let the model attend to multiple reference frames, learning better correspondence to reconstruct the current frame. By augmenting the self-supervised approach with an online adaptation module, our method successfully alleviates tracker drifts caused by spatio-temporal discontinuity. Experimental results show that the effectiveness of proposed modules and our method can bridge the gap between self-supervised and supervised learning. In addition, we train a powerful model with as few as 100 raw videos, indicating that supervised learning or semantic representation is not of the essence for dense tracking on semi-VOS.