基于Hadoop平台的数据分析和应用

Data Analysis and Application Based on Hadoop Platform

传统朴素贝叶斯算法基于独立分类方式,难以满足实际应用需求,针对这个问题,基于Hadoop分布式系统数据管理平台,提出了一种基于加权分析法的朴素贝叶斯改良算法。通过引入加权分析法对属性值进行加权处理,采用相关系数法和相关概率法来保证权重系数选择的合理性,有效提高分类精确度。实例结果表明:改良的加权朴素分析法在进行大规模数据测量中,具备很高的分类准确率和较快的分类速度,但在测试小数据样本时不能很好的体现优势,因此,该算法在大数据分析中具备了很高的数据挖掘分类优势。

This paper aims at the problem that the traditional naive Bayesian algorithm is difficult to meet the practical application requirements because of its independent classification,and puts forward an improved naive Bayesian algorithm based on weighted analysis method.By introducing the weight analysis method to attributed value weighted,the correlation coefficient method and the correlation probability method are adopted to ensure the rationality and classification accuracy of the weight coefficient.The example results show that the improved method has high classification accuracy and fast classification speed in large-scale data measurement,but it cannot be very good in testing small data samples.Therefore,the algorithm has a high advantage of data mining classification in big data analysis.