在线-离线数据流上复杂事件检测

Complex Event Processing over Live Archived Data Streams

随着数据采集和处理技术的发展,在物联网对象跟踪、网络监控、金融预测、电信消费模式等领域中进行事件检测显得越发重要.事件检测在一次扫描数据流的假设下完成,数据流在被处理完后丢弃.事实上,很多应用场景中,历史数据流因含有丰富的信息而不能简单丢弃,且一些事件检测查询需要同时在实时和历史数据流上进行.鉴于已有复杂事件检测很少考虑同时在实时-历史数据流上进行模式匹配,作者研究了在线-离线数据流上复杂事件检测的关键问题.主要工作如下:(1)针对滑动窗口内产生的大量模式匹配中间结果,提出利用时态关系和时空关系管理中间结果的方法 TPM和STPM.STPM以中间结果的时态和状态信息为权值对中间结果进行管理,将最近的、最有可能更新状态的中间结果置于内存,极大地减少了中间结果的读取操作代价.(2)给出了基于选择度的在线-离线复杂事件检测优化算法;(3)给出了算法的复杂性分析和代价模型;(4)在基于时空关系的中间结果管理模型下,在一个在线-离线复杂事件检测原型系统中进行实验,对多个参数(子窗口大小,选择度,匹配率,命中率)进行了算法对比分析.实验结果充分验证了所提出的算法的可行性和高效性.

With the development of data collection and data processing techniques,event detection has become increasingly vital in application areas such as object-tracking in IOT,network monitoring,financial prediction,and telecommunication consumption mode detection,etc.Event processing is supposed to be completed in one-pass of the data streams which are discarded after pattern matching.Actually,historical streams maintain plentiful information which cannot be simply discarded in many scenarios and some event detection queries are always subscribed over both live and archived（historical） streams.Due to the lackness of event processing over live and archived event streams,this paper addresses key issues of live-archived stream complex event processing.Main works are as follows：（1）Due to large numbers of partial matches generated in a sliding window,partial matches management methods named TPM and STPM are proposed.With STPM,spatial and temporal information are kept into partial matches and the most recent and possible updated partial matches are resided in main memory which can reduce pattern match miss ratio and greatly alleviate external partial match loading I/O cost.（2） Optimization of complex event processing algorithm over live-archived streams based on events selectivity is proposed.（3）Formal cost model of related methods are presented.（4） Based on the proposed partial matches management methods,extensive performance comparison experiments in a prototype CEP system are evaluated（experimental parameters include subwindow size,selectivity,match ratio,hit ratio,etc）.Experimental analysis verifies soundness and effectiveness of the proposed methods.