生成式人工智能大型语言模型在消化道癌症领域辅助科研创作的现状分析:基于2024年美国临床肿瘤学会中国学者数据

A comprehensive analysis of large language models in generative artificial intelligence-assisted research writing:insights from 2024 ASCO gastrointestinal oncology data by Chinese scholars

目的利用2024年中国学者于美国临床肿瘤学会(ASCO)发表的摘要论文,描述大型语言模型(LLMs)在肿瘤研究领域辅助科研创作的现状。方法收集中国学者(包含港澳台地区)2024年在ASCO发表的关于消化道癌症研究领域的摘要论文305篇,通过GPTZero Deep Learning检测生成式人工智能(Gen AI)生成概率。2021年未出现Gen AI工具时中国学者60篇论文作为阴性对照,并使用Gen AI生成的论文作为阳性对照,受试者工作特征(ROC)曲线评价GPTZero Deep Learning检测Gen AI生成内容的准确性,Pearson相关分析验证Gen AI生成概率与人类创作概率相关性,应用总体质量得分评价摘要论文质量。结果所有纳入摘要论文中,按地域排序,北京(50篇)、广东(49篇)和上海(48篇)数量位列前3位。按癌症种类排序,占比最多的前5位依次是肝癌(29.51%)、食管癌(18.69%)、泛癌种(14.43%)、结直肠癌(10.82%)和胃癌(10.82%)。免疫治疗是研究热点,程序性死亡受体-1(PD-1)单抗、程序性死亡配体-1(PD-L1)单抗、细胞毒性T淋巴细胞相关蛋白4(CTLA-4)单抗与PD-1/CTLA-4双抗和PD-1/酪氨酸基抑制基序域(TIGIT)双抗分别占临床试验的77.28%,11.36%,2.27%,6.82%和2.27%。分子靶向药物中酪氨酸激酶抑制剂占85.19%,多激酶抑制剂占11.11%。在ROC曲线最佳阈值下,GPTZero Deep Learning检测Gen AI内容准确性的敏感度和特异度均为100%。GPTZero Deep Learning检测发现,2024年在线发表的摘要论文中包含Gen AI内容的概率高于2021年,差异有统计学意义[17%(6%,35.5%)vs.5.5%(3%,12.75%),P<0.001]。在诊断置信度下的Gen AI生成概率与人类创作概率呈显著负相关(r=-0.852,P<0.001)。东部经济区Gen AI生成概率高于其他省份,非临床研究组高于临床研究组,差异有统计学意义(P<0.05)。人类创作摘要的总体质量得分要显著高于AI生成,差异有统计学意义[(13.7±1.8)分vs.(8.9±2.2)分,P<0.001)]。结论与2021年相比,2024年中国学者ASCO摘要中Gen AI内容的信号显著增加,人类创作摘要的总体质量要优于Gen AI。

Objective To delineate the current role of large language models(LLMs)in facilitating scientific research in oncology based on abstracts presented by Chinese scholars at ASCO in 2024.Methods 305 abstract papers on gastrointestinal cancer research published by Chinese scholars(including Hong Kong,Macao and Taiwan)at ASCO in 2024 were collected,and the generation probability of generative artificial intelligence(Gen AI)was detected by GPTZero Deep Learning.In 2021,60 papers of Chinese scholars without Gen AI tools were used as negative controls,and papers generated by Gen AI were used as positive controls.ROC curve evaluated the accuracy of GPTZero Deep Learning in detecting content generated by Gen AI.Pearson correlation analysis was used to verify the correlation between Gen AI generation probability and human creation probability,and the overall quality score was used to evaluate the quality of abstract papers.Results Among all abstracts included,Beijing(50 papers),Guangdong(49 papers)and Shanghai(48 papers)ranked the top three by region.The top five cancers were liver cancer(29.51%),esophageal cancer(18.69%),pan-cancer(14.43%),colorectal cancer(10.82%)and stomach cancer(10.82%).Immunotherapy remains the hottest topic,with PD-1,PD-L1,CTLA-4,PD-1/CTLA-4 and PD-1/TIGIT accounting for 77.28%,11.36%,2.27%,6.82%and 2.27%respectively.In molecular targeted drugs,tyrosine kinase inhibitors account for 85.19%,while multi-kinase inhibitors account for 11.11%.Under the optimal threshold of ROC curve,both the sensitivity and specificity of GPTZero Deep Learning to detect the accuracy of Gen AI content were 100%.GPTZero Deep Learning found that the probability of online abstract papers containing Gen AI content was higher in 2024 than that in 2021,with a statistically significant difference[17%(6%,35.5%)vs.5.5%(3%,12.75%),P<0.001].The probability of Gen AI generation under diagnostic confidence was negatively correlated with the probability of human creation(r=-0.852,P<0.001).The generation probability of Gen AI in eastern economic regions was higher than that in other provinces,and that in non-clinical study group was higher than that in clinical study group,with a statistically significant difference(P<0.05).The overall quality score of human-created abstracts was significantly higher than that of Gen AI generated,and the difference was statistically significant[(13.7±1.8)scores vs.(8.9±2.2)scores,P<0.001)].Conclusion Compared with 2021,the signal of AI content in ASCO abstracts of Chinese scholars in 2024 increased significantly,and the overall quality of human-created abstracts was better than that of Gen AI.