中外文搜索引擎自然语言问答能力的比较与评价研究

Comparison and Evaluation of Natural Language Question Answering Capability of Chinese and Foreign Language Search Engines

【目的/意义】对Google,Bing,百度和搜狗四个中外文搜索引擎的自然语言问答能力进行评价,以揭示搜索引擎正在向兼具搜索和自动问答功能的系统演进的趋势,对不同搜索引擎在不同类型问题上的自然语言回答能力进行比较【方法/过程】从文本检索会议和自然语言处理与中文计算会议的问答系统评测项目抽取了三类问题(人物类、时间类、地点类),并进行搜索,以搜索引擎是否返回准确答案或包含正确答案的精选摘要为标准进行人工评分,使用单因素方差分析和多重比较检验的方法进行比较分析。【结果/结论】主流的中外文搜索引擎均已具备一定的自然语言问答能力,但仍存在较大的提升空间。Google总体表现最好,但对于人物类问题的回答能力弱于搜狗.中外文搜索引擎在时间类问题上的表现均好于人物类和地点类问题.

[ PuqDose/significance ] This study aim to evaluate the natural language question-answering (QA) ability of four Chinese and foreign search engines (Google, Bing, Baidu and Sogou), revealing the trend that search engines are evolving to systems with both searching and automatic question-answering functions. [Method/process] We have compared the QA ability of four different search engines on three different types of questions. This study has extracted three types of natural language questions (Who- question, When- question and Where- question) from QA evaluation project of the Text Retrieval Conference(TREC) and the Natural Language processing and Chinese Computing Conference(NLPCC). Each question has been searched by four search engines separately and the results have been rated manually by the standard whether the search engine returns an accurate answer or a featured abstract with the correct answer. One-way ANOVA and LSD tests were used to compare and analyze QA ability among four different search engines. [ Result/conclusion ]The mainstream Chinese and foreign search engines all have some natural language QA ability, but still have to be promoted greatly. Google's overall performance is the best, but the ability to answer who- questions is weaker than that of Sogou. The four search engines all perform better on when- questions than on who- and where- questions.