基于FastDFS的数字媒体系统设计与实现技术研究

Research on Design and Implementation of Digital Media System Based on FastDFS

海量的数据存储一般采用分布式文件系统HDFS,Hbase等作为存储工具,但对海量媒体资产数据这种中小文件的存储,存在性能瓶颈,并且随着后期小文件越来越多,会造成系统元数据占用过大。对于海量媒体资产管理而言,检索是关键,设计良好的查询体系很重要。结合国家重点研发课题海量数字媒体与应用的实际应用需求,通过分析分布式文件系统FastDFS的原理与缺陷,提出了一种基于容器技术、适合于存储海量中小型文件的分布式存储系统,同时设计了集群服务器之间的同步目录组方案,确保在宕机的情况下,用户的数据不会丢失,且Docker支持多租户,确保了不同租户数据的隔离性。最后对设计实现的原型系统进行了测试与分析,测试结果表明,基于FastDFS的数字媒体资产在线管理系统符合预期设计,性能和可靠性较好,可适用于超算环境下各种渲染任务的使用。

Distributed file systems such as HDFS and Hbase are commonly used as storage tools to store massive data.However,there exists bottlenecks in the storage of medium and small files for massive media asset data,and as small files increase,system metadata may occupy too much space.For mass media asset management,retrieval is the key,and well-designed query system is very important.Combined with actual application requirements of huge amounts of digital media and application of the national key project,based on the analysis of the principles and defects of distributed file system FastDFS,we propose a distributed storage system based on container technology which is suitable for storing large and medium-sized files.At the same time,we design a synchronized directory group scheme between cluster servers to ensure that users’ data will not be lost in case of downtime.Docker supports multi-tenancy and can ensure the isolation of different tenant data.Finally,the prototype system designed and implemented is tested and analyzed.The test results show that the digital media asset online management system based on FastDFS is in line with the expected design with better performance and reliability,which is applicable for various rendering tasks in a supercomputing environment.