配电网SCADA时序数据集群的RWI快速查询技术

RWI Fast Query Technology for Time Series Data Cluster of Distribution Network SCADA

针对配电网大量调度监控准实时数据查询效率不高的问题,利用富网络组件容器和大数据二级索引机制将配电大数据嵌入到大规模并行处理（massively parallel processor,MPP）查询引擎中,提出一种跨平台的配电网数据RWI快速查询新方法。综合运用Impala数据守护进程,实现大量准实时数据在调度监控应用的快速查询。以铁路10k V配电网监控系统工程导出的数千万级实际时序数据为算例,进行加载测试和集群查询性能测试。结果表明：基于二级索引的RWI方法异步回调机制,在正常运行下的集群磁盘I/O读取速度约为存储速度的10倍,能将大数据集群与监控界面端异步回调接口间的数据延迟降至数百ms级,合理提高集群性能,能够适当地提升海量数据响应能力,但远低于扩大集群节点数对海量数据响应能力的提升效果。

According to a large number of distribution dispatching and monitoring data in real time query efficiency is not high, the rich network component container and large data two level index of big data distribution mechanism will be embedded into the massively parallel processor（MPP） query engine, proposed a cross platform support distribution network dispatching and monitoring RWI large data query method. The Impala data daemon was used to realize the rapid query of a large number of quasi real time data in the server side of the scheduling monitoring. Taking the tens of millions of real time series data derived from the railway 10 k V distribution network monitoring system as an example, the loading test and the cluster query performance test were carried out. The results show that the RWI method of two level index of asynchronous callback mechanism based on 10 times under normal operation of cluster disk I/O read speed is about storage speed, can be a big data cluster and monitoring interface end asynchronous callback interface between data delay to hundreds of MS level, improve the reasonable cluster performance, can improve the mass data response ability, but far less than the number of nodes in cluster expansion response ability to enhance the effect of massive data.