基于HBase的工业时序大数据分布式存储性能优化策略

Performance optimization strategy of distributed storage for industrial time series big data based on HBase

在自动化的工业场景中,大量工业设备产生的时序性日志数据量呈爆炸式增长,业务场景对时序数据的访问需求进一步提升。虽然目前基于分布式列族的数据库HBase能够存储工业时序大数据,但由于未考虑特定业务场景中数据与访问行为特征的关联,现有策略无法较好地满足工业时序数据的特定访问需求。针对上述问题,基于分布式存储系统HBase,利用工业场景中数据与访问行为特征的关联性,提出面向海量工业时序数据的分布式存储性能优化策略。针对工业时序数据特点引发的负载倾斜问题,提出基于冷热数据分区及访问行为分类的负载均衡优化策略。使用逻辑回归模型(LR)对数据进行冷热分类,并将热数据分散存储在不同节点;同时,为进一步降低存储集群中跨节点的通信开销,以提升工业时序数据高维索引的查询效率,提出索引主数据同Region化策略,设计索引RowKey字段及拼接规则,将索引存放到与它对应的主数据相同的Region中。在真实工业时序数据上的实验结果表明,引入优化策略后的数据负载分布倾斜度降低28.5%,查询效率提升27.7%,验证了所提优化策略能够有效地挖掘特定时序数据的访问模式,合理地分配负载,降低数据访问开销,有能力满足对特定时序大数据的访问需求。

In automated industrial scenarios,the amount of time series log data generated by a large number of industrial devices has exploded,and the demand for access to time series data in business scenarios has further increased.Although HBase,a distributed column family database,can store industrial time series big data,the existing strategies cannot meet the specific access requirements of industrial time series data well because the correlation between data and access behavior characteristics in specific business scenarios is not considered.In view of the above problem,based on the distributed storage system HBase,and using the correlation between data and access behavior characteristics in industrial scenarios,a distributed storage performance optimization strategy for massive industrial time series data was proposed.Aiming at the load tilt problem caused by characteristics of industrial time series data,a load balancing optimization strategy based on hot and cold data partition and access behavior classification was proposed.The data were classified into cold and hot ones by using a Logistic Regression(LR)model,and the hot data were distributed and stored in different nodes.In addition,in order to further reduce the cross-node communication overhead in storage cluster and improve the query efficiency of the high-dimensional index of industrial time series data,a strategy of putting the index and main data into a same Region was proposed.By designing the index RowKey field and splicing rules,the index was stored with its corresponding main data in the same Region.Experimental results on real industrial time series data show that the data load distribution tilt degree is reduced by 28.5%and the query efficiency is improved by 27.7%after introducing the optimization strategy,demonstrating the proposed strategy can mine access patterns for specific time series data effectively,distribute load reasonably,reduce data access overhead,and meet access requirements for specific time series big data.