基于SolrCloud的分布式科技项目查重系统

A SolrCloud-based Distributed System Checking For Duplicate Science & Technology Projects

从海量科技项目文件库检测出与待检测科技项目的相似文本,设计能够支持实时响应的可扩展的科技项目查重系统。首先采用分布式计算对科技项目文件进行预处理,建立全文倒排索引;然后在分布式SolrCloud系统上执行相似性计算,采用标题段落语句模型(TPSM)计算待检测科技项目文本与全库科技项目文本的相似度。搜狗实验室约10万篇规模文本集上的测试结果表明:所提出的标题段落语句模型(TPSM)调和平均值比全文向量空间模型(FVSM)高出约15%,比段落空间模型(PVSM)高出8%左右;在检测性能上,一篇1 000字左右的待检测论文检测时间约为10 s,达到实时检测需求。实验结果同时表明,基于SolrCloud的分布式系统的扩展性、容错性均能满足实际需求。

To check the duplicated science and technolog projects from the massive documents library, a SolrCloud- based distributed system is designed ＆ developed, it is scalable ＆ can respond in real time. The system first processes projects using distributed computing, establishes a full-text inverted index, then pefformes similarity computation on a distributed SolrCloud platform and the tide-paragraph-sentence model （TPSM）. Test on the Sogou laboratory dataset including about 100 thousand documents shows that, the proposed title-paragraph-sentence model （TPSM）＇s harmonic mean value is about 15% higher than that of the tiff1 text vector dpace model （FVSM）, and is about 8% higher than the paragraph space model （PVSM）. In the speed testing on 1 000 documents, the system running time is about 10 seconds, satisfying the user＇s real-time running needs. The experimental results show that, the SolrCloud -based distributed system can well meet the planned requirements.