基于多维度和多模态信息的视频描述方法

Video description method based on multidimensional and multimodal information

针对视频自动描述任务中的复杂信息表征问题,提出一种多维度和多模态视觉特征的提取和融合方法。首先通过迁移学习提取视频序列的静态和动态等多维度特征,并采用图像描述算法提取视频关键帧的语义信息,完成视频信息的特征表征;然后采用多层长短期记忆网络融合多维度和多模态信息,最终生成视频内容的语言描述。实验仿真表明,所提方法与目前已有方法相比,在视频自动描述任务中取得了较好的效果。

In order to solve the problem of complex information representation in automatic video description tasks, a multi-dimensional and multi-modal visual feature extraction and fusion method was proposed. Firstly, multi-dimensional features such as static and dynamic attributes of the video sequence were extracted by transfer learning, and the image description algorithm was also used to extract the semantic information of the key frames in the video. By doing this, the video features extraction was carried out. Then, multi-layer long and short memory networks were used to fuse multi-dimensional and multi-modal information, and finally generated a language description of the video content. Compared with the existing methods, experimental simulations results show that the proposed method achieves better results in the video automatic description task.