目的构建一种基于人工智能大语言模型(large language model,LLM)技术、可用于医学教育的新型虚拟患者(virtual patient,VP)系统,评价该系统在基层医生进修学习全科医学临床思维中的应用效果。方法选取2021年1月至2024年2月在东南大学...目的构建一种基于人工智能大语言模型(large language model,LLM)技术、可用于医学教育的新型虚拟患者(virtual patient,VP)系统,评价该系统在基层医生进修学习全科医学临床思维中的应用效果。方法选取2021年1月至2024年2月在东南大学附属中大医院进修的基层社区医生为研究对象,随机分为试验组和对照组,分别采用基于LLM的VP系统教学、传统教学方法进行授课,通过临床思维理论知识考核、临床思维能力考核、课程满意度调查评估教学效果,并对结果进行相应的统计学分析。结果共纳入124名基层社区医生,其中试验组60例、对照组64例,两组在一般基线资料上差异无统计学意义,具有可比性。课程结束后,试验组临床思维理论知识考核成绩显著高于对照组(83.83±3.15 vs.79.92±4.52,P<0.01),且不及格率显著低于对照组(0.00%vs.9.38%,P<0.05);试验组在临床思维能力3个维度(批判性、系统性、循证思维)方面教学后分数均显著高于教学前,而对照组仅在批判性思维维度上教学前后差异有统计学意义;教学后试验组在系统思维、循证思维方面分数均显著高于对照组(P<0.05),但在批判性思维上两组分数差异无统计学意义。试验组对授课的总体满意度也显著高于对照组(93.33%vs.85.48%,P<0.05)。结论基于LLM的VP系统提升了学员对临床思维理论知识的掌握程度,也促进了其临床思维能力的培养,该教学方法可为其他医学教育群体提供新的教学工具和思路。展开更多
BACKGROUND Small intestinal bacterial overgrowth(SIBO)poses diagnostic and treatment challenges due to its complex management and evolving guidelines.Patients often seek online information related to their health,prom...BACKGROUND Small intestinal bacterial overgrowth(SIBO)poses diagnostic and treatment challenges due to its complex management and evolving guidelines.Patients often seek online information related to their health,prompting interest in large language models,like GPT-4,as potential sources of patient education.AIM To investigate ChatGPT-4's accuracy and reproducibility in responding to patient questions related to SIBO.METHODS A total of 27 patient questions related to SIBO were curated from professional societies,Facebook groups,and Reddit threads.Each question was entered into GPT-4 twice on separate days to examine reproducibility of accuracy on separate occasions.GPT-4 generated responses were independently evaluated for accuracy and reproducibility by two motility fellowship-trained gastroenterologists.A third senior fellowship-trained gastroenterologist resolved disagreements.Accuracy of responses were graded using the scale:(1)Comprehensive;(2)Correct but inadequate;(3)Some correct and some incorrect;or(4)Completely incorrect.Two responses were generated for every question to evaluate reproducibility in accuracy.RESULTS In evaluating GPT-4's effectiveness at answering SIBO-related questions,it provided responses with correct information to 18/27(66.7%)of questions,with 16/27(59.3%)of responses graded as comprehensive and 2/27(7.4%)responses graded as correct but inadequate.The model provided responses with incorrect information to 9/27(33.3%)of questions,with 4/27(14.8%)of responses graded as completely incorrect and 5/27(18.5%)of responses graded as mixed correct and incorrect data.Accuracy varied by question category,with questions related to“basic knowledge”achieving the highest proportion of comprehensive responses(90%)and no incorrect responses.On the other hand,the“treatment”related questions yielded the lowest proportion of comprehensive responses(33.3%)and highest percent of completely incorrect responses(33.3%).A total of 77.8%of questions yielded reproducible responses.CONCLUSION Though GPT-4 shows promise as a supplementary tool for SIBO-related patient education,the model requires further refinement and validation in subsequent iterations prior to its integration into patient care.展开更多
文摘目的构建一种基于人工智能大语言模型(large language model,LLM)技术、可用于医学教育的新型虚拟患者(virtual patient,VP)系统,评价该系统在基层医生进修学习全科医学临床思维中的应用效果。方法选取2021年1月至2024年2月在东南大学附属中大医院进修的基层社区医生为研究对象,随机分为试验组和对照组,分别采用基于LLM的VP系统教学、传统教学方法进行授课,通过临床思维理论知识考核、临床思维能力考核、课程满意度调查评估教学效果,并对结果进行相应的统计学分析。结果共纳入124名基层社区医生,其中试验组60例、对照组64例,两组在一般基线资料上差异无统计学意义,具有可比性。课程结束后,试验组临床思维理论知识考核成绩显著高于对照组(83.83±3.15 vs.79.92±4.52,P<0.01),且不及格率显著低于对照组(0.00%vs.9.38%,P<0.05);试验组在临床思维能力3个维度(批判性、系统性、循证思维)方面教学后分数均显著高于教学前,而对照组仅在批判性思维维度上教学前后差异有统计学意义;教学后试验组在系统思维、循证思维方面分数均显著高于对照组(P<0.05),但在批判性思维上两组分数差异无统计学意义。试验组对授课的总体满意度也显著高于对照组(93.33%vs.85.48%,P<0.05)。结论基于LLM的VP系统提升了学员对临床思维理论知识的掌握程度,也促进了其临床思维能力的培养,该教学方法可为其他医学教育群体提供新的教学工具和思路。
文摘BACKGROUND Small intestinal bacterial overgrowth(SIBO)poses diagnostic and treatment challenges due to its complex management and evolving guidelines.Patients often seek online information related to their health,prompting interest in large language models,like GPT-4,as potential sources of patient education.AIM To investigate ChatGPT-4's accuracy and reproducibility in responding to patient questions related to SIBO.METHODS A total of 27 patient questions related to SIBO were curated from professional societies,Facebook groups,and Reddit threads.Each question was entered into GPT-4 twice on separate days to examine reproducibility of accuracy on separate occasions.GPT-4 generated responses were independently evaluated for accuracy and reproducibility by two motility fellowship-trained gastroenterologists.A third senior fellowship-trained gastroenterologist resolved disagreements.Accuracy of responses were graded using the scale:(1)Comprehensive;(2)Correct but inadequate;(3)Some correct and some incorrect;or(4)Completely incorrect.Two responses were generated for every question to evaluate reproducibility in accuracy.RESULTS In evaluating GPT-4's effectiveness at answering SIBO-related questions,it provided responses with correct information to 18/27(66.7%)of questions,with 16/27(59.3%)of responses graded as comprehensive and 2/27(7.4%)responses graded as correct but inadequate.The model provided responses with incorrect information to 9/27(33.3%)of questions,with 4/27(14.8%)of responses graded as completely incorrect and 5/27(18.5%)of responses graded as mixed correct and incorrect data.Accuracy varied by question category,with questions related to“basic knowledge”achieving the highest proportion of comprehensive responses(90%)and no incorrect responses.On the other hand,the“treatment”related questions yielded the lowest proportion of comprehensive responses(33.3%)and highest percent of completely incorrect responses(33.3%).A total of 77.8%of questions yielded reproducible responses.CONCLUSION Though GPT-4 shows promise as a supplementary tool for SIBO-related patient education,the model requires further refinement and validation in subsequent iterations prior to its integration into patient care.
