一个基于词语搭配的英文文本检索软件的实现

AN IMPLEMENTATION OF ENGLISH TEXT RETRIEVAL SOFTWARE BASED ON WORD COLLOCATION

词语搭配是英语语言学研究中的一个重要课题,近年来,趋向于注重数据验证和量化研究。探讨一个基于搭配研究的英文文本检索软件ColloStu实现的关键技术。该软件设计一个通配符匹配算法,该算法使用确定有限自动机DFA,通过压缩自动机的状态数,加快匹配速度。同时,在检索搭配词时能识别共现语境中的句子终结符,从而更有效地检索出搭配词。对搭配力计算的Z分值算法进行了改进,综合运用Z分值、T分值、MI值从多个角度计算搭配强度,使计算更加准确。实验表明,与主流的检索软件相比,Collo Stu除增加了搭配力计算功能外,单词统计和搭配词检索更准确。

Word collocation is an important subject in the study of English linguistics. In recent years,it tends to focus on data validation and quantitative research. This paper discusses the key technology of Collo Stu,an English text retrieval software based on collocation research. The software designs a wildcard matching algorithm that uses the DFA to speed up the matching speed by compressing the number of its states. It can identify the sentence terminator in the cooccurrence context in order to retrieve the collocations more effectively. We have improved the Z score algorithm of collocation calculation. We use Z score,T score and MI value to compute collocation intensity from multiple angles to make the calculation more accurate. Experiments show that,compared with the mainstream search software,Collo Stu addition to adding the collocation calculation function,its word statistics and collocation word search is more accurate.