整体评分与分项评分的再思考——二语语音能力评测方法

Rethinking Holistic Scoring and Analytic Scoring——Rating Scale on L2 Pronunciation Competence

"整体评分"和"分项评分"是语言评测中常见的评分方式,而评分方式的选择会直接影响评分信度。该研究首先通过回顾和对比国内外英语作为二语/外语的标准化口试中的语音评分标准及描述语,分析其蕴含的评分方式,然后结合对评分员的问卷调查和访谈,以及国内外评分员语音评分的统计分析,探究我国英语标准化考试中应采用的评分方式和评分员应具备的素养。研究发现,国外口试语音部分的评分标准属于"整体+分项评分",国内口试语音部分的评分标准属于"整体评分",但两者均在不同程度上需要依赖评分员的语音知识和主观判断进行评分;评分员在"整体评分"中,因评分构念差异大,致使评分员间信度弱,而"分项评分"中评分员间信度及内部一致性均显著提升。据此,该研究提出我国英语口试中的语音评分应采用"分项评分"或具化描述语,引导评分员提高评测客观性和针对性。

The holistic scoring and analytic scoring are two common rating methods in language assessment,and the choice of them directly affects the rating reliability.In pronunciation rating,holistic scoring has been widely applied,but analytic scoring is gaining more and more attention.Comparative research on these two rating methods is in need.This paper aims at exploring a more suitable method in rating pronunciation in standardized EFL speaking tests in China,in association with the phonetic knowledge Chinese raters should have.First,a comparative study was carried out to investigate the pronunciation rating methods in domestic(CET,TEM)and international(TOEFL,IELTS,GESE)standardized speaking tests.The rating criteria and descriptors concerning pronunciation in these ESL/EFL tests were analyzed and compared.Second,a survey on pronunciation rating constructs of Chinese CET and TEM raters was conducted,and an empirical study was carried out to compare rater reliability when holistic and analytic scoring methods were adopted in the pronunciation rating.570 Chinese students’speeches were first scored holistically by three Chinese raters and four native English-speaking raters.Canonical correlations between the holistic ratings by different raters were analyzed.Next,the 570 speeches were scored by the three Chinese raters on five dimensions of pronunciation(namely segmental accuracy),fluency,stress,intonation and clarity.40 speeches were scored repeatedly to check intra-rater reliability.Pearson’s correlation test was carried out between the ratings by individual raters and the mean ratings of the three raters on each of the five analytic scales.The findings showed that there were both similarities and differences in the pronunciation rating criteria in domestic and international ESL/EFL speaking tests.Both were concise,and attached great importance to segmental accuracy,pausing,and intelligibility or communication.However,the pronunciation rating criteria in international ESL/EFL speaking tests combined both holistic and analytic scoring,and the descriptors often concerned the relationship between different pronunciation dimensions,while those in domestic EFL speaking tests were mainly based on holistic scoring,with very general descriptors,and different dimensions of pronunciation were often described separately.Also,holistic rating criteria concerning intelligibility and suprasegmental features in international ESL/EFL speaking tests reflect a dependence on a native ear and English phonetic knowledge.This seems to suggest reliability problems when holistic criteria were applied by Chinese raters,since raters differed greatly in their rating constructs when scoring pronunciation holistically.There was a low correlation between the holistic ratings in either the Chinese or native English-speaking rater groups.However,when pronunciation was rated analytically,the intra-rater reliability was high,and the inter-rater reliability on all analytic scales was significantly higher than that for the holistic ratings,except for the clarity scale,as raters differed in their understanding of clarity.The findings suggest that in rating pronunciation,holistic scoring might be less reliable than analytic scoring,as pronunciation encompasses different dimensions,and raters tend to have different rating constructs and put different weights on various dimensions.Considering that the raters of standardized EFL speaking tests in China are mostly Chinese EFL teachers,who are varied in their English phonetic knowledge,it might be a better choice to adopt analytic scoring in rating pronunciation,or use more specific descriptors on both segmental and suprasegmental dimensions of pronunciation in the rating criteria,so as to improve reliability and discrimination.