流数据Top-K关键字查询算法

Algorithm for Top-K Keyword Query in Data Streams

基于Spark Streaming计算框架的分布式Top-K关键字查询是统计流数据中所有关键字的热点研究问题。多数研究通过限定存储空间来实现Top-K关键字查询,并假设关键字集合已知。针对这个问题,提出一种可应用于关键字集合未知情况的分布式Top-K关键字查询算法,根据监测到的关键字动态地调整存储空间,通过更新策略的优化提升其精度。实验结果表明,该算法的性能在关键字集合未知的情况下比现有算法更优。

Distributed Top-K keyword query based on the framework of Spark Streaming is a hot research issue. It is used to count all keywords in data streams. Most studies of Top-K keyword query limit storage space and assume that the keywords set is known. To solve this problem, we presented a distributed Top-K keyword query algorithm which can be used in cases where the keywords set is unknown. This algorithm dynamically adjusts the size of storage space according to monitored keywords and optimizes the updated strategy to improve precision. Experimental results show that the proposed algorithm under the condition of unknown keywords set has better performance.