基于随机平衡采样的不确定大数据流在线分类算法

Uncertain big data flow online classification algorithmbased on random balanced sampling

不确定大数据流具有动态性和不平衡性特点,导致分类结果不精准,为此提出基于随机平衡采样的不确定大数据流在线分类算法。根据Spark框架大数据筛选结构,过滤筛选不确定大数据,释放不满足条件的大数据。应用Hoeffding算法计算实际值与观测值之差,判断差值与属性差值之间的关系,确定最佳分类属性。随机设定最小类别与最大类别数目,对非平衡样本抽样,避免因样本规模过大而造成的样本损失。建立在线分类器,逐级筛选不确定大数据。通过计算不确定信息增益,获取归类最大可能性类别,实现大数据流的在线分类。由实验结果可知,该算法对数据集W1、W2、W3分类的准确率最低值分别达到90%、94%、83%,具有精准分类效果。

Uncertain big data flows are dynamic and unbalanced,which results in inaccurate classification results.Therefore,an uncertain big data flow online classification algorithm based on random balanced sampling is proposed.According to the big data filtering structure of Spark framework,the uncertain big data is filtered and released,and the unqualified big data is released.The Hoeffding algorithm is used to calculate the difference between the actual value and the observed value,judge the relationship between the difference value and the attribute difference value,and determine the best classification attribute.The minimum and maximum number of categories are set randomly.The unbalanced samples are sampled to avoid the sample loss caused by the excessive sample size.Online classifiers are established to screen out uncertain big data step by step.By calculating the gain of uncertain information,the maximum possible classification category is obtained to classify the online big data flows.According to the experimental results,the classification accuracy of the algorithm for data sets W1,W2 and W3 can reach 90%,94%and 83%,respectively,so the algorithm has an accurate classification effect.