采用概化理论改善临床技能测试设计的研究

Utilizing the generalizability theory to improve the clinical skills assessment design

目的采用概化理论探讨如何降低临床技能测试设计的误差及提高信度。方法选择2018年广州医科大学附属第六医院住院医师规范化培训第二年的40名住院医师,通过腹腔穿刺、气管插管、心脏体格检查、清创缝合术与胸腔穿刺术5个考站来测试其相关的临床技能,每站2位评分者,通过概化研究计算临床测试成绩各个向面的变异数及其向面解释百分率,进一步通过决策研究计算代表临床技能测试设计信度的指标--概化系数和独立指标值。概化系数和独立指标值越高,临床技能测试设计的信度越高。结果采用概化理论计算后得出总变异数为268.016,向面解释百分率分别为研究对象39.9%(106.936/268.016),考站6.8%(18.190/268.016),评分者0.0%(0.000/268.016),研究对象和考站41.9%(112.395/268.016),研究对象和评分者0.0%(0.000/268.016),考站和评分者1.8%(4.838/268.016)和无法解释因素9.6%(25.747/268.016)。当增加测试考站或评分者时,其概化系数和独立指标值会随着增加;直到达到某一个点时,再增加考站或评分者,概化系数和独立指标值增加的速度会降低。结论欲获得信度最高的形成性评价应设计8个考站,每个考站1位评分者;欲获得信度最高的终结性评价应设计12个考站,每个考站2位评分者。

Objective Generalizability theory was utilized to determine optimal clinical skills assessment designs, which reduce errors and improve reliability.Methods Forty resident doctors acted as the study participants which were in their second year of residency training at the Sixth Affiliated Hospital of Guangzhou Medical University in 2018. Items in the clinical skills assessment included abdominocentesis, tracheal intubation, cardiac physical examination, debridement and suturing, and thoracentesis. Two scorers per station were calculated and analyzed using generalizability theory.Results The results from the multivariate analysis of variance and generalizability theory showed that the total variance was 268.106 and the variance component interpretation percentages were 39.9% (research subject, 106.936/268.106), 6.8% (assessment station, 18.190/268.106), 0.0% (scorer, 0.000/268.106), 41.9% (research subject and assessment station, 112.395/268.106), 0.0% (research subject and scorer, 0.000/268.106), 1.8% (assessment station and scorer, 4.838 /268.106), and 9.6% (residual, research subject and assessment station and scorer, 25.747/268.106). When adding test stations or scorers, their generalization coefficient and independent index values (these two indicators represent the reliability value of the test) increase until a certain point is reached;at that point, by adding test stations or scorers, the rate of increase of the generalization coefficient and the independent index value decrease.Conclusions To obtain the most reliable formative evaluation, eight test stations should be designed, with one scorer at each test station;12 credential stations should be designed for the most reliable final evaluation, with two scorers at each test station.