语料库短语序列提取系统的设计与开发

Design and Implementation of the Phraseological Sequence Extraction System from Large-scale Corpora

语料库短语序列提取一直是短语学研究的关键技术环节。囿于计算和操作的复杂性,前人研究多使用相对单一的统计方法测量和提取短语序列,导致提取的数据包含大量噪音。文章使用前沿的大数据处理手段和计算技术,实现了基于频数、互信息、边界熵等多种统计手段的短语序列提取方法,并研制开发了相应的系统。实验结果表明,该系统能够在普通计算机上支持千万词级规模的大型语料库运算,并能显著提高短语序列的提取质量。

The extraction of phraseological sequences from corpora has become one of the research hotspots in recent years, but due to the computational complexity, previous studies often used a single measurement method to extract the phraseological sequence, and their experimental results also constantly contained a lot of disturbing sequences. In this paper, by using the state-of-the-art big data processing method, we design a new extraction method based on frequency, mutual information and maximum boundary entropy, and we also develop a phraseological sequence extraction software The experimental results show that the software can support computing on a large-scale corpus of tens of millions of word tokens in the ordinary computer, and meanwhile it can reach a higher extraction precision of phraseological sequences in terms of both quantity and quality.