基于Flume的XML数据自动收集系统

Automatic XML Data Collection System Based on Flume

针对分布式系统中的数据收集,提出了一种基于Flume的XML数据自动收集系统。根据实际需求,系统设计了一种新型的多层星状拓扑结构,以满足数据收集实时性的要求。该系统能够自动检测指定存储位置的XML数据更新,解析XML文本后存储到HBase数据库中。在测试中,该系统中各机器的平均处理速度能够达到550 Kb/s,系统总处理平均速度能够达到1 650 Kb/s。

For data collection in distributed system,a XML data automatic collecting system based on Flume is put forward.According to actual demands on real-time,a new multilayer star topology structure is designed in the system.The data collection system can detect automatically updates of XML data in specific locations,deserialize XML and store in database of HBase.In experiment,average process speed of any computer of the system can reach 550 Kb/s,and total average process speed can reach 1 650 Kb/s.