Yarn架构下基于GA的Web日志挖掘技术

Web log mining technology using GA based on Yarn

提出了一种面向TB级别日志文件挖掘需求的日志挖掘技术。采用MapReduce编程模型设计并实现了一种基于子种群联姻策略的GA,并部署到Yarn架构上,使Yarn架构与GA有效结合。在算法Partition阶段,采用Round-Robin策略代替原有的哈希方法,使各个子种群基因更加趋向均匀分布,增加了子种群收敛效率和结果准确性,同时均衡各个节点运行负载,提高了系统性能。经测试,应用该技术的挖掘结果平均准确度达到93%以上,效率提升接近33%。

This paper proposed a Web log mining technology orienting the demand of mining for TB-level log files.It designed and implemented a genus of GA which deployed on the Yarn using the MapReduce programing model to combine the GA with the Yarn effectively.In addition,took the Round-Robin strategy instead of the hash method during the Partition stage of Ma-pReduce in order to distribute the gene among the sub populations evenly,increased the populations’rate of convergence and validity of result,and balanced the workload amid all of the nodes at the same time.Finally,the test shows that the rate of the validate result using the technology,which improves the efficiency by 33%,the average accuracy is more than 93%.