基于人工智能的药品投诉类案例信息识别及分类效能评估

Performance Evaluation of Artificial Intelligent Information Identification and Classification Regarding Drug Complaint Cases

目的评估构建的神经网络(NN)模型对药品投诉类案例的信息识别与分类效能,探究人工智能(AI)辅助人工审查分类的可行性。方法将X公司2022年8—10月线上收集的3090条反馈记录导入构建的NN模型,利用各指标对AI的分类效能进行评估并以3名资深药物警戒专员一致判定结果为真实值,对比AI与人工检测的灵敏度、特异度及AUC值;此外,用Kappa分数来评价AI与真实值组的一致性。结果与人工组相比,AI的F1值为90.48%,AUC值为95.20%,表明AI具有较高的分类质量;AI的灵敏度为90.48%,人工组为97.62%,差异无统计学意义(P=0.25);AI组和人工组特异度分别为99.87%和99.31%,差异有统计学意义(P<0.001);AI的Kappa值为0.903,表明AI组与真实值组具有极好的一致性。结论AI的分类效能具有较高的质量,对于投诉类案例的识别表现出较高的灵敏性与特异性,可以为投诉和非投诉案例的快速识别分类提供参考。

Objective To evaluate the performance of a constructed language model and neural network(NN)model in the information identification and classification of drug complaint cases to explore the feasibility of using artificial intelligence(AI)to assist in manual review and classification for the drug complaint cases.Methods A total of 3090 feedback records collected online by Company X from August 2022 to October 2022 were imported into the constructed NN model.Multiple matrices evaluated the classification performance of AI.The consensus results of three senior pharmacovigilance specialists were used as the ground truth value to compare the sensitivity,specificity and AUC value of the AI model and those of manual detection;In addition,the kappa value was used to evaluate the consistency between the AI group and ground truth value group.Results Compared with the manual group,the F1 score and AUC value of the AI group were 90.48%and 95.20%,respectively,which indicates the high quality of the classification of AI.The sensitivity of AI was 90.48%,lower than that of the manual group(97.62%),the difference was not statistically significant(P=0.25);the specificity of the AI group and the manual group were 99.87%and 99.31%,respectively,with significant difference(P<0.001)by statistical analysis;the kappa value of AI group was 0.903,which demonstrates the excellent consistency between AI group and ground truth value group.Conclusion The classification efficacy of AI is of high quality and It exhibits high sensitivity and specificity in identifying complaint cases,which can provide a reference for the rapid identification and classification of complaint cases and non-complaint cases.