摘要
针对海量文本数据处理,为实现快速文本处理响应,缩短海量数据为辅助决策提供服务的时间,基于Hadoop云计算平台,建立HDFS分布式文件系统存储海量文本数据集,通过文本词频利用MapReduce原理建立分布式索引,以分布式数据库HBase存储关键词索引,并提供实时检索,实现对海量文本数据的分布式并行处理。实验结果表明,Hadoop框架为大规模数据的分布式并行处理提供了很好的解决方案。
Aiming at quickly respond to the large-scale text processing, the time of providing service from large-scale data for additional decision should be reduced. Based on Hadoop cloud computing platform, this paper builds distributed file system HDFS to store large-scale text, uses MapReduce method to set up distributed index and distributed database to store the keyword index and provide real-time search,and carries out distributed parallel processing to large-scale text. The experiment results show that Hadoop architecture can provide a better solution for distributed parallel processing of large-scale data.
出处
《无线电通信技术》
2014年第1期54-57,共4页
Radio Communications Technology
基金
国家部委基金资助项目
