基于CSS模板的职位信息并行抽取系统设计

Research on and Implementation of the Parallel Information Extraction System on Recruitment with CSS Template

针对现有职位信息抽取方法由于缺乏自适应性和并行性,存在冗余度高和抽取效率低的问题,提出了基于CSS模板的方式并行职位信息抽取方法。该方法根据职位信息页面特点使用CSS路径抽取方法,并制定抽取模板解决抽取的准确性和自适应性,使用了MapReduce编程模型实现职位信息的并行化抽取。使用MD5算法计算已抽取得到的职位信息的MD5值,结合MapReduce并行计算编程模型的特性实现职位信息去重,最终将去重后的职位信息存储在分布式数据库HBase。实验测试结果表明,并行计算与传统的非并行编程模型相比在处理的时间效率和采集的职位信息量上都有明显的提高。

A parallel position information extraction method based on CSS template is proposed to address the high redundancy and low efficiency due to the lack of adaptability and parallelism by existing position information extraction method. The information extraction that employs the CSS path builds the extraction template according to the structure of the job web page, thus guaranteeing the accuracy and adaptability. The parallel extraction is based on the MapReduce. The MD5 algorithm is used to compute the value of extracted information, and the values are used in the Reduce function to reduce the duplicate position information. Finally, the position information is saved in HBase, a distributed column-oriented database. Experimental results on a set of data show that the proposed approach offers better extraction efficiency and the speed than the non-parallel extraction programming model.