基于分布式数据仓库的分类分析研究

Research of classification analysis for distributed data warehouse

针对GAC-RDB分类算法只能应用于单机版数据仓库的局限性,为了能够更方便、快捷地在云计算平台上开展数据挖掘工作,基于分布式数据仓库HBase,结合GAC-RDB分类算法的实现机理,制定适合分布式平台的运行策略,使用原生HiveQL语言提出了一种分布式GAC-RDB分类算法。实验显示,随着集群中节点的不断增加,算法的运行时间稳步下降。结果表明,在保证算法准确率的前提下,分布式数据仓库能够有效提高GACRDB分类算法的扩展性和运行效率,相对于MapReduce框架,HiveQL语言降低了对数据挖掘从业人员的技术要求,更大程度地减少了算法的开发时间,为挖掘海量数据提供了新的解决方案。

According to the limit of GAC-RDB classification algorithm which was designed for stand-alone data warehouse, in order to carry out data mining works more convenient and efficient on cloud computing platform, based on HBase, a distributed data warehouse, and the implementation mechanism of GAC-RDB classification algorithm, this paper proposed a distributed strategy, put forward the distributed GAC-RDB classification algorithm by native HiveQL language. Experiments show that the algorithm running time steadily decline as increased the number of nodes in the cluster. Results indicate that the efficiency of GAC-RDB algorithm can be improved when it is working on a distributed data warehouse, with extended scalability. Relative to the MapReduce framework, HiveQL cut down the technical requirements for data mining workers, decrease development time of the algorithm.