多项选择题命题技术:基于新手命题员培训的案例

Investigating the Development of Multiple-choice Item Writing Skills:A CaseStudy of an Item-Writing Training Program

聚焦一项语言测评课程中的命题培训,分析27名研究生作为新手命题员,在经过命题知识讲解,独立编写的多项选择题和小组合作修订后,命题成果中的题目缺陷,比较部分新手与专业命题员编写的题目在作答结果上的差异,并对其中9名研究对象进行课后访谈。研究发现,新手命题员的题目经过合作修订后质量有较大提升,在难度和区分度方面部分题目可以达到与专业命题员相似或更高的水平,以实践和合作为基础的命题培训对语言测评素养有提升作用。研究结果为语言测试多项选择题命题技术的学习效果提供了证据,为命题培训提供了参考。

This case study,set in a Chinese as a second language(CSL)teaching environment,examines an item-writing training session focusing on multiple-choice items in a language assessment course for graduate students majoring in applied linguistics or educational evaluation.In the item-writing training process,the 27 participants took lectures on item-writing principles before they generated items of reading and listening comprehension individually with the purpose of measuring CSL proficiency,and then revised and refined them collaboratively in groups.The items were evaluated based on the principles of multiple-choice item writing adapted from Haladyna and Rodriguez(2013)to analyze the quality of items before and after collaborative revision.3 sets of reading comprehension items,two written collaboratively by the novice item writers and one by professional item writers,were piloted on 82 L2 Chinese learners to compare the quality of items in terms of item difficulty,discrimination,and non-functional distractors.Additionally,nine participants were interviewed about their experience and conception of language assessment in the course.The study found an obvious improvement in the quality of items revised collaboratively compared to those generated individually after lecture,with the mean number of item writing flaws dropping from 1.04 to 0.36 per item.Moreover,one of the two sets of items written collaboratively by novice item writers exhibited higher quality compared to the set written by professional item writers because of moderate difficulty and higher discrimination.The interviews revealed that item-writing training was evaluated positively in the language assessment course as it was practice-based and useful in future careers,raising the level of language assessment literacy in various aspects.In short,the evidence was supportive of the effectiveness of the item-writing training program.