突发事件检测的MapReduce并行化实现

Parallel Implementing Bursty Events Detection Using Map Reduce

【目的】在大数据环境下,从文本流中准确且快速地检测出特定领域的突发事件。【方法】利用Kleinberg突发检测方法和LDA主题模型方法,将其扩展到Map Reduce并行框架中,实现并行语料预处理、并行突发词检测、并行突发文档过滤和并行主题提取。【结果】对新闻文本流进行模拟仿真实验,结果表明,该并行方法在特定领域突发事件检测中准确率P、召回率R和调和平均值F分别最高可达87.50%、77.78%和82.35%。【局限】基于Map Reduce的并行方法难以实现大规模动态文本流在线(Online)实时(Real-time)突发事件检测。【结论】与传统串行突发事件检测方法相比,所构建的分布式并行化方法在保证检测结果正确性的同时,具有良好的可扩展性,性能得到较大提升。

[Objective] In big data environment, this paper aims to accurately and quickly detect bursty events from the text stream. [Methods] Using Kleinberg bursty detection and LDA topic model, the method is extended to MapReduce framework to achieve parallel corpus predisposed, parallel detection of bursty word, parallel filtration of bursty document and parallel extraction of topic. [Results] The results of simulation experiments on the news text stream show that precision reaches 87.50%, recall reaches 77.78%, and F-measure reaches 82.35% with the parallel method to detect bursty events in specific areas. [Limitations] The MapReduce parallel method is difficult to achieve Online and Real-time detection ofbursty events with large-scale dynamic text stream. [Conclusions] Compared with the traditional serial detecting method of bursty events, the distributed parallel method not only guarantees the accuracy of detecting results, but also has a good scalability.