基于深度多模态特征融合的短视频分类

Deep multimodal feature fusion for micro-video classification

目前,短视频已经成为新媒体时代极具有代表性的产物之一,其天然的具有时短、强编辑等特点,使得传统视频分类模型不再适合于短视频分类任务。针对综合短视频分类问题的特点,提出了一种基于深度多模态特征融合的短视频分类算法。所提算法将视觉模态信息和音频模态信息输入到域分离网络中,将整个特征空间划分为所有模态共享的公有域部分及由音频模态和视觉模态分别独有的私有域部分,借助优化域分离网络,最大程度地保留了不同模态特征间的差异性和相似性。在公开的短视频分类数据集上进行实验,证明了所提算法可以有效减少特征融合时的冗余性,并将分类的平均精度提高到0.813。

Nowadays,micro-video has become one of the most representative products in the new media era.It has the characteristics of short time and strong editing,which makes the traditional video classification models no longer suitable for micro-video classification task.Based on the characteristics of the micro-video classification problem,the micro-video classification algorithm based on deep multimodal feature fusion is proposed.The proposed algorithm inputs the visual modal information and acoustic modal information into the domain separation network,and divides the entire feature space into a shared domain part shared by all modalities and the private domain part unique to the acoustic and visual modalities respectively.By optimizing the domain separation network,the differences and similarities among different modal features are preserved to the greatest extent.The experiments on the public micro-video classification dataset prove that the proposed algorithm can effectively reduce the redundancy of feature fusion and improve the average classification accuracy to 0.813.