基于Hadoop的分布式视频转码系统研究

Research on Distributed Video Transcoding System Based on Hadoop

当下由于视频内容多样化的爆发式变革,产生了多种音视频封装格式和编码格式,为解决用户高清视频多端下载收看及相应格式转换需求,应对庞大的数据量计算作业,需整合高效计算机资源。该文提出了一种基于Hadoop的分布式视频转码方案,采用分布式文件存储系统HDFS进行大型视频文件的存储,通过MapReduce编程框架结合FFmpeg开源软件,将视频数据处理划分为Map和Reduce两个阶段,把庞大的数据量分布到多处理节点分析。调用转码模块,减少开发人员工作量,分布式完成视频转码功能。该方案充分利用了数据集群的并行计算能力,突破了单机视频转码技术的发展瓶颈。通过实验验证得出,相比于单一节点进行视频转码,此系统的转码速度仅在2台数据节点的分布式集群中就获得了50%的提升。通过此系统可以为各类终端用户按各自需求提供易于使用、开放便捷、快速高效的视频转码服务。

At present, due to the diversified and explosive changes of video content, a variety of audio and video packaging formats and encoding formats have been produced. In order to meet the needs of users for multi-end downloading and viewing of high-definition videos and corresponding format conversion requirements, it is necessary to deal with huge data volume calculation tasks and integrate efficient computer resource assistance. We propose a Hadoop-based distributed video transcoding scheme, which uses the distributed file storage system HDFS to store large video files. Through the MapReduce programming framework combined with the FFmpeg open source software, the video data processing is divided into two stages: Map and Reduce. Distribute the huge amount of data to multi-processing nodes for analysis. Call the transcoding module to reduce the workload of developers and complete the video transcoding function in a distributed manner. This solution makes full use of the parallel computing capabilities of the data cluster and breaks the bottleneck of the development of stand-alone video transcoding technology. Experimental verification shows that compared to a single node for video transcoding, the transcoding speed of this system is only 50% improved in a distributed cluster of 2 data nodes. Through this system, it is possible to provide various end-users with easy-to-use, open and convenient, fast and efficient video transcoding services according to their needs.