基于递归长短期记忆网络和镜头序列注意网络的视频摘要生成

Video Summarization Generation Based on ReLSTM and Shot-Sequence-Attention

为解决基于长短期记忆网络LSTM的视频摘要生成方法当输入序列过长时LSTM网络中的记忆单元不能集中在长时间序列的跨度上。通过深度学习的方法研究了一种基于递归长短期记忆网络(ReLSTM)和序列注意(sequential attention,SSA)的视频摘要生成模型用以提高深度学习网络学习时序特征的能力。该模型使用ReLSTM网络提取时间特征。同时,利用SSA动态调整每个视频序列输入到ReLSTM网络中的特征权重。结果表明:在数据集TVSum上F 1-score平均提高2.5%,最高提高0.2%。在数据集SumMe上F 1-score平均提高7.8%,最高提高3.4%。可见本文方法能有效地学习镜头之间的时序特征。

In the current LSTM-based video summarization generation methods,memory units in the LSTM network cannot be focused on the span of long time series when the input sequence is too long.Based on recurrent long short-term memory network(ReLSTM)and sequential attention(SSA),a video summary generation model was proposed to improve the ability of deep learning networks to learn temporal features.This model uses the ReLSTM network to extract temporal features.Meanwhile,SSA was used to dynamically adjust the feature weights of each video sequence input into the ReLSTM network.The experimental results show that F 1-score is increased by 2.5%on average and 0.2%at the highest on the dataset TVSum.On the dataset SumMe,F 1-score is improved by 7.8%on average and 3.4%at the highest.The experimental results show that this method can effectively learn the temporal features between shots.