Effect of training on resident inter-reader agreement with American College of Radiology Thyroid Imaging Reporting and Data System

BACKGROUND The American College of Radiology Thyroid Imaging Reporting and Data System(ACR TI-RADS)was introduced to standardize the ultrasound characterization of thyroid nodules.Studies have shown that ACR-TIRADS reduces unnecessary biopsies and improves consistency of imaging recommendations.Despite its widespread adoption,there are few studies to date assessing the inter-reader agreement amongst radiology trainees with limited ultrasound experience.We hypothesize that in PGY-4 radiology residents with no prior exposure to ACR TIRADS,a statistically significant improvement in inter-reader reliability can be achieved with a one hour training session.AIM To evaluate the inter-reader agreement of radiology residents in using ACR TIRADS before and after training.METHODS A single center retrospective cohort study evaluating 50 thyroid nodules in 40 patients of varying TI-RADS levels was performed.Reference standard TI-RADS scores were established through a consensus panel of three fellowship-trained staff radiologists with between 1 and 14 years of clinical experience each.Three PGY-4 radiology residents(trainees)were selected as blinded readers for this study.Each trainee had between 4 to 5 mo of designated ultrasound training.No trainee had received specialized TI-RADS training prior to this study.Each of the readers independently reviewed the 50 testing cases and assigned a TI-RADS score to each case before and after TI-RADS training performed 6 wk apart.Fleiss kappa was used to measure the pooled inter-reader agreement.The relative diagnostic performance of readers,pre-and post-training,when compared against the reference standard.RESULTS There were 33 females and 7 males with a mean age of 56.6±13.6 years.The mean nodule size was 19±14 mm(range from 5 to 63 mm).A statistically significant superior inter-reader agreement was found on the post-training assessment compared to the pre-training assessment for the following variables:1.“Shape”(k of 0.09[slight]pre-training vs 0.67[substantial]post-training,P<0.001),2.“Echogenic foci”(k of 0.28[fair]pre-training vs 0.45[moderate]post-training,P=0.004),3.‘TI-RADS level’(k of 0.14[slight]pre-training vs 0.36[fair]post-training,P<0.001)and 4.‘Recommendations’(k of 0.36[fair]pre-training vs 0.50[moderate]post-training,P=0.02).No significant differences between the preand post-training assessments were found for the variables'composition','echogenicity'and'margins'.There was a general trend towards improved pooled sensitivity with TI-RADS levels 1 to 4 for the post-training assessment while the pooled specificity was relatively high(76.6%-96.8%)for all TI-RADS level.CONCLUSION Statistically significant improvement in inter-reader agreement in the assigning TI-RADS level and recommendations after training is observed.Our study supports the use of dedicated ACR TI-RADS training in radiology residents.