视频云存储服务场景下的HDFS负载均衡工具

HDFS Load Balancer for Video Cloud Storage Service

在线视频服务是互联网服务的重要内容,存储是在线视频服务提供的基础.HDFS作为面向通用文件的云存储系统,被很多视频服务网站采用,但其负载均衡工具没有考虑利用视频文件在线播放时的带宽消耗特性使集群的带宽资源得到更充分的利用.为解决这一问题,提出视频存储场景下的负载均衡方法 VOBM,它对视频文件在线播放时的带宽消耗与视频文件的码率、数据块大小和访问热度的关系进行了分析并建立了新的负载评估模型,在此基础上它在负载方案生成和负载调度两个环节中加入了对带宽消耗因素的考虑.在HDFS原有负载均衡工具的基础上实现了该方法,实验证明该方法能够有效避免高带宽消耗数据块的聚集,在高带宽消耗视频文件作为服务访问热点的实验场景中,该方法在90%的场景中优于原有负载均衡方法,最高能使数据节点集群中瓶颈节点的带宽峰值降低20%.

Online video service is an important part of internet service, and storage is the basis of it. As a cloud storage system aiming at storing common files, HDFS is applied by many video service websites,but its load balancer doesn＇t take videos＇ characteristics of bandwidth consuming when played online into consideration to make full use of bandwidth resource of cluster. In order to solve this problem, proposes a load balancing method called VOBM under video storage circumstance, which anaylzes the relationship of band- width consuming of videos when played online between their bitrate,size of block and popularity and built up a new load evaluating model, then on the basis of it this method takes factor of bandwidth consuming into consideration in generating plans and scheduling blocks. Implemented VOBM on the basis of original load balancer in HDFS. Experiments show that VOBM can effectively avoid clus- tering of high bandwidth consuming data blocks, and it outperforms original balancing method in 90% of cases and manages to decrease bandwidth peak of bottleneck datanode of cluster by 20% in the best case when high bandwidth consuming files act as hot spots in the experiments.