结合多尺度注意力机制和双向门控循环网络的视频摘要模型

Video summarization model based on the multiscale attention mechanism and bidirectional gated recurrent network

任务中全局注意力在长距离视频序列上注意力值分布的方差较大,生成关键帧的重要性分数偏差较大,且时间序列节点边界值缺乏长程依赖导致的片段语义连贯性较差等问题,通过改进注意力模块,采用分段局部自注意力和全局自注意力机制相结合来获取局部和全局视频序列关键特征,降低注意力值的方差。同时通过并行地引入双向门控循环网络(bidirectional recurrent neural network,BiGRU),二者的输出分别输入到改进的分类回归模块后再将结果进行加性融合,最后利用非极大值抑制(non-maximum suppression,NMS)和核时序分割方法(kernel temporal segmentation,KTS)筛选片段并分割为高质量代表性镜头,通过背包组合优化算法生成最终摘要,从而提出一种结合多尺度注意力机制和双向门控循环网络的视频摘要模型(local and global attentions combine with the BiGRU,LG-RU)。该模型在TvSum和SumMe的标准和增强数据集上进行了对比试验,结果表明该模型取得了更高的F-score,证实了该视频摘要模型保持高准确率的同时可鲁棒地对视频完成摘要。

In the video summary task,the variance of global attention value distribution on long distance video sequences is large,the importance score of generating key frames is large,and the semantic coherence of fragments is poor due to the lack of long-range dependence on the boundary values of time series nodes.Herein,by improving the attention module,segmented local self-attention and global self-attention mechanisms are merged to acquire the key features of local and global video sequences and lower the variance of attention values.Concurrently,the bidirectional gated recurrent neural network(BiGRU)is introduced in parallel,the output is input into the enhanced classification regression module,and afterward,the results are additively fused.Lastly,nonmaximum suppression and kernel temporal segmentation methods are applied to filter fragments and segment them into high-quality representative shots.The final summary is created by the knapsack combinatorial optimization algorithm.The video summary model LG-RU,which integrates the multiscale attention mechanism and BiGRU,is developed and compared with TvSum and SumMe’s standard and enhanced data sets.It is demonstrated that the model has a higher F-score,which verifies that this model can complete the video summary robustly while preserving high accuracy.