基于Spark的时态查询扩展与时态索引优化研究

Research on Temporal Query Expansion and Temporal Index Optimization Based on Spark

时空数据库和基于集群计算的时间分析工具大多基于外存,将其应用在大数据处理场景下系统性能将迅速降低。为此,基于Spark构建一个易用且高可扩展的时态大数据查询分析系统。通过扩展Spark SQL解析器,使其能够支持类SQL形式的时态操作,运用SIMBA开源项目的方法,引入全局过滤和局部时态索引2种优化策略,使得系统能以高吞吐量及低延迟执行时态查询操作。基于时态查询效率的评估实验结果表明,在不同影响参数下,该系统的时态查询性能优于原生的Spark SQL查询处理方案。

There exists some temporal databases and temporal analysis tools based on cluster-based computing systems.However,most of them are disk-oriented and performance degenerate rapidly when processing big data. This paper proposes a system which is based on Spark,and provides accessible and scalable temporal query scheme with large temporal data for users. Specifically,it extends Spark SQL parser to support SQL-like temporal operations. Besides,it uses the index manager based on Spark SQL which is proposed by SIMBA,and embeds optimization strategies in two aspects：global filtering and local temporal index. Depending on these optimization rules,the system achieves high throughput and lowlatency in temporal operations. Evaluation experiment results on temporal query efficiency and effectiveness showthis system has improved temporal query performance over original Spark SQL in different factors.