危化品领域专业分词库构建与应用研究

Research on Construction and Application of Professional Corpus in the Field of Hazardous Chemicals

现有通用分词库无法在危化品领域专业文本中取得良好的分词效果,因此采用结合机器学习分词结果与人工筛选整理的方式提取专业语料,设计包含词表、词条及词条属性的分词库分级架构,提出基于标识树结构的词库特征向量索引方法,形成危化品领域专业分词库,并将该分词库作为自定义词典应用于分词模型中进行验证。结果表明,与通用分词库相比,专业分词库能够提升危化品领域文本分词准确率,有助于对危化品领域文档数据进行深化分析。

Given that the current professional corpus in the field of hazardous chemicals cannot achieve good analysis results in the professional text of hazardous chemicals,the professional corpus was extracted by combining machine learning algorithm and manual screening and sorting.The hierarchical structure of thesaurus including thesaurus,term and term attributes were designed.A feature vector index method based on identification tree structure was proposed to form a professional thesaurus in the field of hazardous chemicals,and the thesaurus was applied to the word segmentation model as a custom dictionary for verification.Experimental results showed that,compared with the general corpus,the proposed corpus can greatly improve the word segmentation accuracy of the text in the field of hazardous chemicals,and lay a foundation for the in-depth analysis and utilization of document data.