异种语料融合方法:基于统计的中文词法分析应用

A Method of Merging Corpora in Different Annotation Standards:An Application on Statistics Chinese Lexical Analysis

基于统计的中文词法分析往往依赖大规模标注语料,语料的规模和质量直接影响词法分析系统的性能。高覆盖率、高质量的语料资源非常有限,而且适用于不同领域的语料往往具有不同的分词和词性标注标准,难以直接混合使用,从而导致既有资源未能充分利用,分词精度下降等问题。针对该问题,该文提出了简单有效的异种语料的自动融合方法,并通过实验验证了提案方法的有效性、较强的实用性以及对多种语料融合的可扩展性。

Large scale manually annotated corpora are usually used in research on statistical Chinese lexical analysis.The scale and quality of corpora affect the performance of statistical lexical analysis directly.Corpora in high quality and high rate of coverage are very valuable but limited,and it is very difficult to combine corpora of different domains directly since they are different in segmentation and part of speech（POS） tagging standards.These problems make it difficult to utilize existing resources and prevent the performance improvment in Chinese lexical analysis.To address this issue,this paper presents a simple but effective strategy to optimize the performance and domain adaptability of Chinese lexical analysis by merging different domains＇ corpora automatically.Our experiments verify the validity,the stronger practicability,and the scalability to multiple corpora of the proposed method.