自然语言处理评测中的问题与对策

Problems and Countermeasures in Natural Language Processing Evaluation

自然语言处理中的评测任务引导和推动着技术、模型和方法上的研究。近年来,新的评测数据集和评测任务不断被提出,与此同时,现有评测暴露的一系列问题也限制了自然语言处理技术的进步。该文从自然语言处理评测的概念、构成、发展和意义出发,分类综述了主流自然语言处理评测的任务和特点,进而总结归纳了自然语言处理评测中的问题及其成因。最后,该文参照人类语言能力评测规范,提出类人机器语言能力评测的概念,并从信度、难度、效度三个方面提出了一系列类人机器语言能力评测的基本原则和实施设想,并对评测技术的未来发展进行了展望。

Evaluation in natural language processing drives and promotes research on models and methods.In recent years,new evaluation data sets and evaluation tasks have been continuously proposed.At the same time,a series of problems exposed by such evaluations seems to restrict the progress of natural language processing technology.Starting from the concept,composition,development and significance of natural language Processing evaluation,this article classifies and summarizes the tasks and characteristics of mainstream natural language Processing evaluation,and then reveals the problems and their possible causes.In parallel to the human language ability evaluation standard,this paper puts forward the concept of human-like machine language ability evaluation,and proposes a series of basic principles and implementation ideas for human-like machine language ability evaluation from three aspects:reliability,difficulty and validity.