Low rating reliability has long been the primary concern in school-based oral English achievement tests. In this study, a computer-aided rating system （CARS） was developed to improve inter- and intra-rater reliabili...Low rating reliability has long been the primary concern in school-based oral English achievement tests. In this study, a computer-aided rating system （CARS） was developed to improve inter- and intra-rater reliability through the instantiation of rating criteria, task division and random distribution, on-line training, reliability verification and sound wave ＂reading＂ and ＂writing.＂ A rating experiment was conducted among six raters to compare intra- and inter-rater reliability between traditional rating and rating with CARS. At the end of each round of rating, a conference was held. Both quantitative an~ qualitative analyses show that CARS can significantly improve inter- and intra-rater reliability, mainly through helping raters use criteria more accurately and focus more attention on rating. In addition, the research has also shed light upon further study on improving rating reliability.展开更多
文摘Low rating reliability has long been the primary concern in school-based oral English achievement tests. In this study, a computer-aided rating system （CARS） was developed to improve inter- and intra-rater reliability through the instantiation of rating criteria, task division and random distribution, on-line training, reliability verification and sound wave ＂reading＂ and ＂writing.＂ A rating experiment was conducted among six raters to compare intra- and inter-rater reliability between traditional rating and rating with CARS. At the end of each round of rating, a conference was held. Both quantitative an~ qualitative analyses show that CARS can significantly improve inter- and intra-rater reliability, mainly through helping raters use criteria more accurately and focus more attention on rating. In addition, the research has also shed light upon further study on improving rating reliability.
