搜索引擎用短语词典建设

Constructing phrase dictionary for search engine

百度搜索引擎分词系统中的原短语库是由统计得出,含有大量噪音。委托北京大学计算语言学研究所进行了手工整理标注。为此设计了加工规范,对短语进行了严格的定义,制定了加工操作手册,并开展了手工工程,通过严格的控制保证标注的一致性。共整理短语119 984条,噪音短语比例约占7%。当前该短语词典已应用于百度搜索引擎中,对提高检索效果和减少索引空间都起到一定作用。

There are many phrase noises in the Baidu search engine system. These phrases are got by statistical method. So Baidu company consigns the computational linguistics institute of Peking University to purify them by hand. In order to complete the project, the processing specification is designed. After formal defining the concept of phrase and designing the operation manual, the language resource project is carried out. Many measures are used for guaranteeing the consistency. 119 986 phrases are tagged, and a...