面向分布式环境的分层数据采集技术研究

Research on the hierarchical data collection technologyfor a distributed environment

目的:研究使用基于Flume的流式数据分层采集技术,以提高数据采集的可靠性与有效性,为流计算引擎提供稳定的数据流。方法:使用基于Flume的流式数据分层采集模型,并在此基础上提出分层采集进程监控方法和基于分组的双层哈希负载均衡方法。结果:从数据迁移率、吞吐量、节点平均耗时三个指标进行评价,实验结果表明,运用基于分组的双层哈希负载均衡方法能有效地均衡节点间的负载,提高数据采集效率。结论:分布式环境下,运用基于Flume的流式数据分层采集模型,所提的分层采集进程监控方法和双层哈希负载均衡方法可有效提高数据采集的稳定性与可靠性,因而可为流计算引擎提供实时、稳定的数据流。

Aims:The technology of hierarchical data collection based on Flume was used to improve the reliability and effectiveness of data transmission and provide a stable flow for the streaming computing engine.Methods:The method of layered process monitor and an algorithm of packed-based double layers of hash load balancing were used based on Flume.Results:The experimental results showed that the model increased the effectiveness of data transmission and effectively balanced the load according to the index of the migration rate,the throughput and the consuming time.Conclusions:In a distributed environment,a hierarchical data acquisition model based on Flume and a double-layer hash load balancing method can increase the reliability of data transmission,thus providing the real-time and stable data flow for streaming computing engines.