分布式流数据加载和查询技术优化

Optimization on Distributed Stream Data Loading and Querying

分布式流查询是一种基于数据流的实时查询计算方法,近年来得到了广泛的关注和快速发展。综述了分布式流处理框架在实时关系型查询上取得的研究成果;对涉及分布式数据加载、分布式流计算框架、分布式流查询的产品进行了分析和比较;提出了基于Spark Streaming和Apache Kafka构建的分布式流查询模型,以并发加载多个文件源的形式,设计内存文件系统实现数据的快速加载,相较于基于Apache Flume的加载技术提速1倍以上。在Spark Streaming的基础上,实现了基于Spark SQL的分布式流查询接口,并提出了自行编码解析SQL语句的方法,实现了分布式查询。测试结果表明,在查询语句复杂的情况下,自行编码解析SQL的查询效率具有明显的优势。

Distributed stream query is a kind of real-time query computation method based on data stream, which has been widely concerned and developed rapidly in recent years. This paper summarized the research results of the distribu- ted stream processing framework in real-time relational query. There is an in-depth comparison of some products, inclu- ding the distributed data loading framework, distributed stream computing framework and distributed stream query sys- tems. The paper proposed a distributed stream query model based on Spark Streaming and Apache Kafka, and designed a fast data loading technology based on virtual memory file system, which gets the data loading speed one time faster compare to Apache Flume. On the basis of Spark Streaming, a distributed stream query interface based on Spark SQL was realized,and a method for parsing SQL queries was proposed to implement distributed query in data stream. The experiment results demonstrate that, in the case of complex SQL queries, the method of analyzing SQL by writing code by oneself has obvious advantages.