基于差分注意力的时空小波分析视频预测算法

Spatial-Temporal Wavelet Analysis Video Prediction Based on Differential Attention Mechanism

针对视频预测中空间结构信息细节和时序运动依赖关系难以准确预测的问题,受人类视觉过程的启发,提出一种基于差分注意力机制的时空小波分析视频预测算法.首先利用时空小波分析模块对视频内容进行多频分解,增强模型对于高频细节信息以及过程性运动的理解能力;然后利用差分注意力机制指导模型更高效、合理地分配注意力资源,提升对瞬时运动特征的表达能力.在KTH, Cityscapes, BAIR, KITTI, Caltech Pedestrian数据集上的实验结果表明,所提算法在PSNR, SSIM, LPIPS评价指标上取得了比已有算法更优异的效果;同时,可视化的对比也表明所提算法的预测结果更加清晰.

Inspired by the visual process of human, a video prediction algorithm based on spatial-temporal wavelet analysis and differential attention is proposed to solve the problem that it is difficult to accurately predict the details of spatial structure information and the dependence of temporal motion. Firstly, the spatial-temporal wavelet analysis module is used to decompose the video in multiple frequencies, so as to enhance the model’s ability to understand high-frequency details and procedural motion. Then, the differential attention mechanism guides the model to allocate attention resources more efficiently and reasonably, and improves the expression ability of instantaneous motion. Experimental results on the KTH, Cityscapes, BAIR, KITTI, Caltech Pedestrian datasets show the proposed algorithm achieves better results than the existing algorithms in the quantitative evaluation metrics of PSNR, SSIM and LPIPS. Meanwhile, the visualization results also show that the prediction of the proposed algorithm is clearer.