Video Frame Prediction by Joint Optimization of Direct Frame Synthesis and Optical-Flow Estimation

Video prediction is the problem of generating future frames by exploiting the spatiotemporal correlation from the past frame sequence.It is one of the crucial issues in computer vision and has many real-world applications,mainly focused on predicting future scenarios to avoid undesirable outcomes.However,modeling future image content and object is challenging due to the dynamic evolution and complexity of the scene,such as occlusions,camera movements,delay and illumination.Direct frame synthesis or optical-flow estimation are common approaches used by researchers.However,researchers mainly focused on video prediction using one of the approaches.Both methods have limitations,such as direct frame synthesis,usually face blurry prediction due to complex pixel distributions in the scene,and optical-flow estimation,usually produce artifacts due to large object displacements or obstructions in the clip.In this paper,we constructed a deep neural network Frame Prediction Network(FPNet-OF)with multiplebranch inputs(optical flow and original frame)to predict the future video frame by adaptively fusing the future object-motion with the future frame generator.The key idea is to jointly optimize direct RGB frame synthesis and dense optical flow estimation to generate a superior video prediction network.Using various real-world datasets,we experimentally verify that our proposed framework can produce high-level video frame compared to other state-ofthe-art framework.

Real-Time Mosaic Method of Aerial Video Based on Two-Stage Key Frame Selection Method

A two-stage automatic key frame selection method is proposed to enhance stitching speed and quality for UAV aerial videos. In the first stage, to reduce redundancy, the overlapping rate of the UAV aerial video sequence within the sampling period is calculated. Lagrange interpolation is used to fit the overlapping rate curve of the sequence. An empirical threshold for the overlapping rate is then applied to filter candidate key frames from the sequence. In the second stage, the principle of minimizing remapping spots is used to dynamically adjust and determine the final key frame close to the candidate key frames. Comparative experiments show that the proposed method significantly improves stitching speed and accuracy by more than 40%.

Flow-aware synthesis: A generic motion model for video frame interpolation

A popular and challenging task in video research,frame interpolation aims to increase the frame rate of video.Most existing methods employ a fixed motion model,e.g.,linear,quadratic,or cubic,to estimate the intermediate warping field.However,such fixed motion models cannot well represent the complicated non-linear motions in the real world or rendered animations.Instead,we present an adaptive flow prediction module to better approximate the complex motions in video.Furthermore,interpolating just one intermediate frame between consecutive input frames may be insufficient for complicated non-linear motions.To enable multi-frame interpolation,we introduce the time as a control variable when interpolating frames between original ones in our generic adaptive flow prediction module.Qualitative and quantitative experimental results show that our method can produce high-quality results and outperforms the existing stateof-the-art methods on popular public datasets.

Deepfake Video Detection Employing Human Facial Features

Deepfake technology can be used to replace people’s faces in videos or pictures to show them saying or doing things they never said or did. Deepfake media are often used to extort, defame, and manipulate public opinion. However, despite deepfake technology’s risks, current deepfake detection methods lack generalization and are inconsistent when applied to unknown videos, i.e., videos on which they have not been trained. The purpose of this study is to develop a generalizable deepfake detection model by training convoluted neural networks (CNNs) to classify human facial features in videos. The study formulated the research questions: “How effectively does the developed model provide reliable generalizations?” A CNN model was trained to distinguish between real and fake videos using the facial features of human subjects in videos. The model was trained, validated, and tested using the FaceForensiq++ dataset, which contains more than 500,000 frames and subsets of the DFDC dataset, totaling more than 22,000 videos. The study demonstrated high generalizability, as the accuracy of the unknown dataset was only marginally (about 1%) lower than that of the known dataset. The findings of this study indicate that detection systems can be more generalizable, lighter, and faster by focusing on just a small region (the human face) of an entire video.

Rate-distortion optimized frame dropping and scheduling for multi-user conversational and streaming video

We propose a Rate-Distortion (RD) optimized strategy for frame-dropping and scheduling of multi-user conversa- tional and streaming videos. We consider a scenario where conversational and streaming videos share the forwarding resources at a network node. Two buffers are setup on the node to temporarily store the packets for these two types of video applications. For streaming video, a big buffer is used as the associated delay constraint of the application is moderate and a very small buffer is used for conversational video to ensure that the forwarding delay of every packet is limited. A scheduler is located behind these two buffers that dynamically assigns transmission slots on the outgoing link to the two buffers. Rate-distortion side information is used to perform RD-optimized frame dropping in case of node overload. Sharing the data rate on the outgoing link between the con- versational and the streaming videos is done either based on the fullness of the two associated buffers or on the mean incoming rates of the respective videos. Simulation results showed that our proposed RD-optimized frame dropping and scheduling ap- proach provides significant improvements in performance over the popular priority-based random dropping (PRD) technique.

Video Inter-Frame Forgery Identification Based on Consistency of Correlation Coefficients of Gray Values

Identifying inter-frame forgery is a hot topic in video forensics. In this paper, we propose a method based on the assumption that the correlation coefficients of gray values is consistent in an original video, while in forgeries the consistency will be destroyed. We first extract the consistency of correlation coefficients of gray values (CCCoGV for short) after normalization and quantization as distinguishing feature to identify interframe forgeries. Then we test the CCCoGV in a large database with the help of SVM (Support Vector Machine). Experimental results show that the proposed method is efficient in classifying original videos and forgeries. Furthermore, the proposed method performs also pretty well in classifying frame insertion and frame deletion forgeries.

基于低秩与稀疏分解的VideoSAR散射关键帧提取

视频合成孔径雷达(video synthetic aperture radar,VideoSAR)的超长相干孔径观测使得区域动态信息的快速浏览极其困难。为以机器视觉方式自动捕捉地物散射消失-瞬态持续-消失-瞬态持续-消失的关键帧变化全过程,提出了一种子孔径能量梯度(subaperture energy gradient,SEG)和低秩与稀疏分解(low-rank plus sparse decomposition,LRSD)相结合的VideoSAR关键帧提取器。提取器为系列性通用架构,适用于任何SEG和LRSD系列方法相结合的形式。所提技术首要针对同时单通道、单波段、单航迹等有限信息条件的解决途径,有助于打破应急响应场景中难以采集多通道、多波段、多航迹或多传感器数据的应用局限性。基于实测数据处理和多种先进LRSD算法进行了对比验证,其代表性散射信息的充分提取可促进未来快速地理解并浓缩区域动态。

基于国密算法的视频媒体文件加密效率提升技术

随着计算机网络和无线通信等技术的发展,有关视频媒体文件的版权保护和信息安全问题日益成为人们关注的焦点,对视频媒体文件加密是一种有效保护信息安全的方式,传统的视频文件加密方法需要对视频媒体文件中所有的视频帧数据进行加密,文件加密的效率较低,加密过程比较耗时.本文针对H.264/AVC视频帧的结构特点,提出了一种基于国产SM2算法的视频媒体文件加密效率提升的方法,该方法在加密视频媒体文件的过程中只加密视频数据中关键帧的NALU Header信息,在检测到H.264分片的情况下同时也需要对non-IDR Header信息进行加密.实验结果表明该方法可以在有效加密视频媒体文件的同时减少了加密所需的时间,明显提升了视频媒体文件的加密效率.

锚框校准和空间位置信息补偿的街道场景视频实例分割

街道场景视频实例分割是无人驾驶技术研究中的关键问题之一,可为车辆在街道场景下的环境感知和路径规划提供决策依据.针对现有方法存在多纵横比锚框应用单一感受野采样导致边缘特征提取不充分以及高层特征金字塔空间细节位置信息匮乏的问题,本文提出锚框校准和空间位置信息补偿视频实例分割(Anchor frame calibration and Spatial position information compensation for Video Instance Segmentation,AS-VIS)网络.首先,在预测头3个分支中添加锚框校准模块实现同锚框纵横比匹配的多类型感受野采样,解决目标边缘提取不充分问题.其次,设计多感受野下采样模块将各种感受野采样后的特征融合,解决下采样信息缺失问题.最后,应用多感受野下采样模块将特征金字塔低层目标区域激活特征映射嵌入到高层中实现空间位置信息补偿,解决高层特征空间细节位置信息匮乏问题.在Youtube-VIS标准库中提取街道场景视频数据集,其中包括训练集329个视频和验证集53个视频.实验结果与YolactEdge检测和分割精度指标定量对比表明,锚框校准平均精度分别提升8.63%和5.09%,空间位置信息补偿特征金字塔平均精度分别提升7.76%和4.75%,AS-VIS总体平均精度分别提升9.26%和6.46%.本文方法实现了街道场景视频序列实例级同步检测、跟踪与分割,为无人驾驶车辆环境感知提供有效的理论依据.

基于互信息量均方差提取关键帧的激光视频图像检索研究

为保证激光视频图像检索结果中不存在重复性冗余图像,提出了基于互信息量均方差提取关键帧的激光视频图像检索方法。基于互信息量均方差的关键帧提取方法,以激光视频图像颜色的互信息量均方差最大化,为激光视频图像关键帧的聚类中心设置标准,以此聚类提取不重复的视频图像关键帧;通过基于关键帧的激光视频图像检索方法,将所提取关键帧作为激光视频图像检索的核心判断内容,提取与所需图像关键帧相似度显著的激光视频图像,完成激光视频图像检索。实验结果显示:此方法使用后,提取的激光视频图像关键帧冗余度仅有0.01,激光视频图像检索结果的MAP指标测试值高达0.98,检索结果中不存在重复性冗余图像。

基于深度学习的视频修复方法综述

视频作为常见的媒体信息之一,目前已在各个领域得到广泛应用。尤其是以抖音等为代表的短视频软件的兴起,使得与视频相关的技术不断迭代更新。其中,视频修复技术是视频处理研究中的一个热点。视频修复技术是利用视频帧内的像素信息和帧间的时域参考信息对视频帧受损的区域进行内容推理并修复,在补全缺损视频、物体移除及视频伪造检测等场景中具有广泛应用前景。该技术可追溯到二十世纪末的老电影修复技术,该任务通常由专业的技术团队逐帧修复完成。而随着数字技术的发展,近年来已有一些人工智能技术用于视频修复,让老电影重获新生。目前,视频修复技术可分为传统方法和基于深度学习的两类方法。其中,传统方法由于缺少对高层语义信息的理解,在场景复杂、缺失区域较大的情况下修复效果不佳;而基于深度学习的方法随着算法框架的优化和图形处理器性能的提升展现了出色的效果,对修复结果的语义结构准确性和时间一致性都有明显的提升。本文在简要回顾传统视频修复方法的基础上,重点分析四类基于深度学习视频修复方法的网络结构、参数模型、性能表现与优缺点;介绍视频修复领域中常用的数据集和评价指标;最后,对视频修复领域现存的问题进行总结并展望未来可能的研究方向。

画框意识:一种视觉媒介要素的历时性演进

边界意识伴随着视觉艺术的成熟而来,它以种种不同的画框形式统治了视觉艺术的历史。理所当然地,物质性画框的普遍存在催生了创作中的画框意识。这种画框意识影响到了从传统绘画到先锋视觉艺术的整个历史过程,同时,也将摄影术、电影和电子游戏纳入到画框意识的话语体系中。从绘画到电子游戏,画框形式从静态演变为动态,而电子游戏则发展出了一套相当复杂且灵活的画框形式。这套画框形式丰富了画框意识的内涵,打破了传统画框结构的封闭性,呈现出了“低张力”的结构特点,而这种画框结构的形成则应归因于电子游戏的互动性本质。

短视频平台公共政策传播的信息框架效应——基于“双减”政策的视频内容分析与实验

[目的/意义]短视频作为新兴的互联网信息表达方式,已逐渐成为政策传播的重要工具。探索短视频平台在公共政策信息传播中的效力以及受众采纳意愿,有助于提升公共政策传播效果。[研究设计/方法]以“双减”政策为例,根据框架效应理论和信息采纳理论,通过内容分析法归纳总结短视频平台中“双减”政策信息框架的组织方式,同时分析不同政策信息框架对传播效果的影响;此外,采用在线控制实验法验证了不同政策信息框架对受众采纳意愿的影响。[结论/发现]在短视频平台中有近四成的信息使用了目标框架,其中损失框架居多,多数采用集体主义视角和非叙事陈述方式;目标框架、文化框架、叙事框架对受众的评论、转发、点赞意愿并未产生显著影响;双减政策信息中,损失、收益框架,集体、个人主义价值观以及叙事和非叙事要素都能增强受众的采纳意愿,并且目标框架与文化框架之间存在交互效应。[创新/价值]研究结论对于政策宣传部门及相关政府机构优化政策传播策略,提高传播效果具有启示作用。

改进视觉Transformer的视频插帧方法

针对现有的视频插帧方法无法有效处理大运动和复杂运动场景的问题,提出了一种改进视觉Transformer的视频插帧方法。该方法融合了基于跨尺度窗口的注意力和可分离的时空局部注意力,增大了注意力的感受野并聚合了多尺度信息;对时空依赖和远程像素依赖关系进行联合建模,进而增强了模型对大运动场景的处理能力。实验结果表明,该方法在Vimeo90K测试集和DAVIS数据集上的PSNR指标分别达到了37.13 dB和28.28 dB,SSIM指标分别达到了0.978和0.891。同时,可视化结果表明,该方法针对存在大运动、复杂运动和遮挡场景的视频能产生清晰合理的插帧结果。

面向视频侦查应用的退化人像GFP深度复原技术

在视频侦查工作中时常遇到视频人像质量过低难以辨识的情况,而GFP等人像深度复原方法只适用于单张人像的复原。为此,本文提出一种基于GFP的监控视频人像复原技术,将GFP方法扩展至视频侦查应用,便于及时锁定犯罪嫌疑人,提高破案效率。首先对监控视频进行视频分帧、人像裁剪对齐、倾斜透视校正等预处理操作,然后使用GFP方法对预处理后得到的人像进行深度复原,最后经过逆处理将复原人像整合成高质量人像视频。在大量经过预处理后的模拟退化人像和无参考退化人像测试集上进行对比实验,结果表明,GFP方法在主观视觉效果和FID、PSFR、SSIM、NIQE等客观量化指标上均优于其他人像深度复原方法,相较于其他人像深度复原方法,能够更有效地复原复杂应用场景下的低质量退化人像,更适用于视频侦查应用;通过使用YTF视频人像数据集进行对比测试实验,结果显示,本文所提出的添加预处理与逆处理过程的基于GFP的监控视频人像复原技术,对于低质量视频人像有更加优秀的复原效果。

瞬态场景下基于光流法的红外视频插帧算法研究

针对现有红外图像插帧方法,在瞬态场景下均不能得到鲁棒性较好的插入帧红外图像,提出了一种基于注意力的多尺度、多分支光流网络,提取相邻2帧红外图像光流信息,每个分支分别学习一种光流信息,利用多尺度特征融合模块在每个尺度上聚焦局部重要信息。设计了一个多光流特征重加权模块,根据通道注意力自适应地选择每个光流的特征。经实验结果证明,所提方法可以很好地完成插帧任务,其性能与最先进的方法相比较更具有优越性。

基于门控时空注意力的视频帧预测模型

针对循环式视频帧预测架构存在精度低、训练缓慢,以及结构复杂和误差累积等问题,提出了一种基于门控时空注意力的视频帧预测模型。首先,通过空间编码器提取视频帧序列的高级语义信息,同时保留背景特征;其次,建立门控时空注意力机制,采用多尺度深度条形卷积和通道注意力来学习帧内及帧间的时空特征,并利用门控融合机制平衡时空注意力的特征学习能力;最后,由空间解码器将高级特征解码为预测的真实图像,并补充背景语义以完善细节。在Moving MNIST、TaxiBJ、WeatherBench、KITTI数据集上的实验结果显示,同多进多出模型SimVP相比,MSE分别降低了14.7%、6.7%、10.5%、18.5%,在消融扩展实验中,所提模型达到了较好的综合性能,具有预测精度高、计算量低和推理效率高等优势。

基于帧间量化参数强度值检测深度伪造人像视频

深度学习算法的发展为伪造人像视频的创作增添了“助力”,应运而生的深度伪造检测成为学界和业界关注的焦点。基于深度学习的伪造视频检测技术需要耗费大量时间与算力训练分类器,深度学习网络的黑盒性和不可解释性也困扰着深度伪造人像视频取证研究。围绕深度伪造人像视频的真实性取证问题,本文提出一种基于视频帧间量化参数强度值的检测方法,并通过二元Logistic回归方程给出参考阈值。实验结果表明:本文的方法在检测利用DeepFaceLab换脸平台合成的深度伪造人像视频中表现出良好的准确率和鲁棒性。所提方法有助于对可疑人像视频真实性进行判断,为涉案人像视频提供取证依据。

基于双参考帧的多源着色结果融合的视频着色方法

对黑白视频着色时,为了更好地利用参考帧信息,提出一种基于双参考帧的多源着色结果融合的着色方法.首先,采用硬注意力融合子模块融合双参考帧提供的颜色信息,以防止双帧语义匹配模块着色时不合理参考信息导致的颜色模糊问题.然后,使用多源着色结果融合模块融合通过双帧光流传播模块、双帧语义匹配模块以及遮挡信息得到的着色结果,进而产生更优质的着色结果.实验结果表明,该方法在Davis30测试集上的峰值信噪比(PSNR)、结构相似性(SSIM)和颜色分布一致性指数(CDC)分别为37.36 dB、0.980 5、0.003 748,说明该方法能够通过多种融合方式充分利用双参考帧的信息为灰度帧着色,并生成美观且具有较好时间一致性的着色结果.

基于计算机视觉和视频插帧的结构振动测试与模态识别

结构振动测试和模态参数识别是基于结构动态特性进行结构健康监测(SHM)的基本方法。为克服传统接触式测试的不便,引入了非接触式计算机视觉方法,其中以智能手机为数据采集装置的方法受到越来越多的关注。然而智能手机相机往往因为性能受限无法满足需求,从而导致测量结果精度下降。为此,提出一种改进的视频插帧算法EQVI-T与改进的边缘检测算法,通过提高原始视频的帧率和提出的特征点追踪方法共同提升计算精度。为验证这一方法的有效性,将其应用于试验室模型的位移响应监测和模态参数识别,并进行了定量和定性评估。结果表明,所提方法在提高测量精度和准确性方面具有显著优势,展示了其在结构振动测试中的潜在应用价值。