基于Spark的大规模文本k-means并行聚类算法

互联网文本数据量的激增使得对其作聚类运算的处理时间显著加长,虽有研究者利用Hadoop架构进行了k-means并行化研究,但由于很难有效满足k-means需要频繁迭代的特点,因此执行效率仍然不能让人满意。该文研究提出了基于新一代并行计算系统Spark的k-means文本聚类并行化算法,利用RDD编程模型充分满足了kmeans频繁迭代运算的需求。实验结果表明,针对同一聚类文本大数据集和同样的计算环境,基于Spark的kmeans文本聚类并行算法在加速比、扩展性等主要性能指标上明显优于基于Hadoop的实现,因此能更好地满足大规模文本数据挖掘算法的需求。

自动化部署工具设计与实现

传统模式下,操作系统、数据库、中间件等常规软件需要人工手动安装,批量部署工作量大,而且各服务器硬件配置不同、应用场景不同、部署方式也各有差异,维护非常具有挑战性,还有就是部署工作具有一定复杂性和重复性。为解决上述问题,设计并开发了一种自动化部署工具,能够支持任务调度、任务编排、脚本维护、一键批量部署等功能,将常规脚本工具化,实现机房设备统一维护管理,极大地降低运维人员的工作压力,提高效率,减少失误率。

Parallel naive Bayes algorithm for large-scale Chinese text classification based on spark

The sharp increase of the amount of Internet Chinese text data has significantly prolonged the processing time of classification on these data.In order to solve this problem,this paper proposes and implements a parallel naive Bayes algorithm(PNBA)for Chinese text classification based on Spark,a parallel memory computing platform for big data.This algorithm has implemented parallel operation throughout the entire training and prediction process of naive Bayes classifier mainly by adopting the programming model of resilient distributed datasets(RDD).For comparison,a PNBA based on Hadoop is also implemented.The test results show that in the same computing environment and for the same text sets,the Spark PNBA is obviously superior to the Hadoop PNBA in terms of key indicators such as speedup ratio and scalability.Therefore,Spark-based parallel algorithms can better meet the requirement of large-scale Chinese text data mining.

云计算PaaS层中POSTGRESQL数据库高可用方案的研究

随着云计算技术在各个领域的不断深化运用,在其PaaS层中各应用的高可用性也成为整个生产环节的重要一环。本文简要论述了当前云计算的主流服务模式,并且以基于Grid Infrastructure为基础设施的云计算平台为例,提出了一种新型的POSTGRESQL数据库的高可用架构。