摘要
当前IKAnalyzer(IK)和ICTCLAS(IC)是主流的中文分词算法。文中首先通过理论对比二者在单机环境下的性能,然后使用Hadoop集群、Hadoop分布式文件管理系统(HDFS)和并行处理大数据集的Map Reduce组成的框架,利用优化后的算法,通过大量的实验对二者在分布式环境下处理大数据集的表现做出比较。
the current IKAnalyzer ( IK ) and ICTCLAS ( IC ) is the mainstream of the Chinese segmentation algorithm. Firstly by theoretical performancecomparison between these two in the single machine environment, and then use the Hadoop cluster, Hadoop distributed file management system ( HDFS ) and parallel processing of large data sets composed of MapReduceframework, using the optimized algorithm, by comparison to make a lot of experiments on the two in the distributed environment for processing large data sets performance.
出处
《网络安全技术与应用》
2014年第12期67-67,71,共2页
Network Security Technology & Application
基金
天津市自然科学基金
合同编号:13JCYBJC16800