海量视频数据分布式存储性能优化方法研究

Research on performance optimization of distributed storage of massive video data

针对文件系统存储效率低、多余副本导致空间浪费以及磁盘能源损耗严重的问题,提出一种新型分布式优化存储策略(distributed optimized storage strategy,DOSS)。首先,引入Bcache混合存储技术在磁盘阵列前增设固态硬盘,作为高速缓冲区对多路视频流进行临时数据组织,变多线程并发写任务为单线程顺序写入任务,规避磁盘内部碎片产生,有效提高系统写入效率。其次,提出改进的liberation码对视频数据进行压缩存储,在保证系统可靠性的同时提高磁盘空间利用率。最后,基于ioctl系统调用编写盘片转速控制函数,实现磁盘多级休眠和低延迟唤醒,减低磁盘能耗,提高使用寿命。结果表明单台存储服务器在500路4 Mbps并发视频流下存储效率提高约36%,存储空间节省约40%,系统应对12000路并发视频流时仍存在休眠磁盘约10%。

This paper proposed a new distributed optimized storage strategy for serious problems of inefficient file system storage,waste of space caused by redundant replicas,and disk energy depletion.First of all,it introduced Bcache hybrid storage technology to add a solid state drive in front of the disk array,as a high-speed buffer to organize the temporary data of multiple video streams,changed the multi-threaded concurrent writing task into a single-threaded sequential writing task,avoided the generation of internal disk fragments,and effectively improved the system writing efficiency.Secondly,it proposed an improved liberation code to compress and store video data,which improved the utilization of disk space while ensuring system reliability.Finally,it wrote disk speed control function based on ioctl system call,realized disk multi-level hibernation and low-latency wake-up,reduced disk energy consumption and increase service life.The results show that the storage efficiency of a single storage server under 500 channel of 4 Mbps concurrent video streams can increase about 36%,save about 40%of storage space,when the system handles 12000 concurrent video streams,there is still about 10%of dormant disks.