Application of artificial intelligence in clinical diagnosis and treatment:an overview of systematic reviews

Objective This study aimed to summarize the characteristics and methodological quality of systematic reviews on the application of artificial intelligence(AI)in clinical diagnosis and treatment.Methods We systematically searched seven English-and Chinese-language literature databases to identify sys-tematic reviews on the application of AI,deep learning,or machine learning in the diagnosis and treatment of any disease published in 2020.We evaluated the methodological quality of the included systematic reviews using“A Measurement tool for the assessment of multiple systematic reviews”(AMSTAR).We also conducted meta-analyses on the diagnostic accuracy of AI on selected disease categories with a large number of included studies and low clinical heterogeneity.Results A total of 40 systematic reviews reporting 1,083 original studies were included,covering 31 diseases from 11 groups of diseases.Eleven systematic reviews were related to neoplasms and nine were systematic reviews related to diseases of the digestive system.We selected digestive system diseases for the meta-analysis.The pooled sensitivities(with 95%confidence interval(CI))of AI to assist the diagnosis of helicobacter pylori,gastrointestinal ulcers,hemorrhage,esophageal tumors,gastric tumors,and intestinal tumors(with 95%CI)were 0.91(0.83-0.95),0.99(0.76-1.00),0.95(0.83-0.99),0.90(0.85-0.93),0.90(0.82-0.95),and 0.93(0.88-0.96),respectively,and the pooled specificities were 0.82(0.77-0.87),0.97(0.86-1.00),1.00(0.99-1.00),0.80(0.71-0.87),0.93(0.87-0.97),and 0.89(0.85-0.92),respectively.The AMSTAR items“the list of included studies”(n=39,97.5%)and“the characteristics of the included studies”(n=39,97.5%)had the highest compliance among the reviews;the compliance was relatively low to the items“the consideration of publication status”(n=1,2.5%),“the consideration of scientific quality”(n=19,47.5%),“data synthesis methods”(n=18,45.0%),and“the evaluation of publication bias”(n=13,32.5%).Conclusions The main subjects of systematic reviews on AI applications in clinical diagnosis and treatment pub-lished in 2020 were diseases of the digestive system and neoplasms.The methodological quality of the systematic reviews on AI needs to be improved,paying particular attention to publication bias and the rigorous evaluation of the quality of the included studies.

Analysis of COVID-19 Guideline Quality and Change of Recommendations:A Systematic Review

Background.Hundreds of coronavirus disease 2019(COVID-19)clinical practice guidelines(CPGs)and expert consensus statements have been developed and published since the outbreak of the epidemic.However,these CPGs are of widely variable quality.So,this review is aimed at systematically evaluating the methodological and reporting qualities of COVID-19 CPGs,exploring factors that may influence their quality,and analyzing the change of recommendations in CPGs with evidence published.Methods.We searched five electronic databases and five websites from 1 January to 31 December 2020 to retrieve all COVID-19 CPGs.The assessment of the methodological and reporting qualities of CPGs was performed using the AGREE II instrument and RIGHT checklist.Recommendations and evidence used to make recommendations in the CPGs regarding some treatments for COVID-19(remdesivir,glucocorticoids,hydroxychloroquine/chloroquine,interferon,and lopinavir-ritonavir)were also systematically assessed.And the statistical inference was performed to identify factors associated with the quality of CPGs.Results.We included a total of 92 COVID-19 CPGs developed by 19 countries.Overall,the RIGHT checklist reporting rate of COVID-19 CPGs was 33.0%,and the AGREE II domain score was 30.4%.The overall methodological and reporting qualities of COVID-19 CPGs gradually improved during the year 2020.Factors associated with high methodological and reporting qualities included the evidence-based development process,management of conflicts of interest,and use of established rating systems to assess the quality of evidence and strength of recommendations.The recommendations of only seven(7.6%)CPGs were informed by a systematic review of evidence,and these seven CPGs have relatively high methodological and reporting qualities,in which six of them fully meet the Institute of Medicine(IOM)criteria of guidelines.Besides,a rapid advice CPG developed by the World Health Organization(WHO)of the seven CPGs got the highest overall scores in methodological(72.8%)and reporting qualities(83.8%).Many CPGs covered the same clinical questions(it refers to the clinical questions on the effectiveness of treatments of remdesivir,glucocorticoids,hydroxychloroquine/chloroquine,interferon,and lopinavirritonavir in COVID-19 patients)and were published by different countries or organizations.Although randomized controlled trials and systematic reviews on the effectiveness of treatments of remdesivir,glucocorticoids,hydroxychloroquine/chloroquine,interferon,and lopinavir-ritonavir for patients with COVID-19 have been published,the recommendations on those treatments still varied greatly across COVID-19 CPGs published in different countries or regions,which may suggest that the CPGs do not make sufficient use of the latest evidence.Conclusions.Both the methodological and reporting qualities of COVID-19 CPGs increased over time,but there is still room for further improvement.The lack of effective use of available evidence and management of conflicts of interest were the main reasons for the low quality of the CPGs.The use of formal rating systems for the quality of evidence and strength of recommendations may help to improve the quality of CPGs in the context of the COVID-19 pandemic.During the pandemic,we suggest developing a living guideline of which recommendations are supported by a systematic review for it can facilitate the timely translation of the latest research findings to clinical practice.We also suggest that CPG developers should register the guidelines in a registration platform at the beginning for it can reduce duplication development of guidelines on the same clinical question,increase the transparency of the development process,and promote cooperation among guideline developers all over the world.Since the International Practice Guideline Registry Platform has been created,developers could register guidelines prospectively and internationally on this platform.

Investigation and evaluation of randomized controlled trials for interventions involving artificial intelligence

Objective Complete and transparent reporting is of critical importance for randomized controlled trials(RCTs).The present study aimed to determine the reporting quality and methodological quality of RCTs for interventions involving artificial intelligence(AI)and their protocols.Methods We searched MEDLINE(via PubMed),Embase,Web of Science,CBMdisc,Wanfang Data,and CNKI from January 1,2016,to November 11,2020,to collect RCTs involving AI.We also extracted the protocol of each included RCT if it could be obtained.CONSORT-AI(Consolidated Standards of Reporting Trials-Artificial Intelligence)statement and Cochrane Collaboration’s tool for assessing risk of bias(ROB)were used to evaluate the reporting quality and methodological quality,respectively,and SPIRIT-AI(The Standard Protocol Items:Recommendations for Interventional Trials-Artificial Intelligence)statement was used to evaluate the reporting quality of the protocols.The associations of the reporting rate of CONSORT-AI with the publication year,journal’s impact factor(IF),number of authors,sample size,and first author’s country were analyzed univariately using Pearson’s chi-squared test,or Fisher’s exact test if the expected values in any of the cells were below 5.The compliance of the retrieved protocols to SPIRIT-AI was presented descriptively.Results Overall,29 RCTs and three protocols were considered eligible.The CONSORT-AI items“title and abstract”and“interpretation of results”were reported by all RCTs,with the items with the lowest reporting rates being“funding”(0),“implementation”(3.5%),and“harms”(3.5%).The risk of bias was high in 13(44.8%)RCTs and not clear in 15(51.7%)RCTs.Only one RCT(3.5%)had a low risk of bias.The compliance was not significantly different in terms of the publication year,journal’s IF,number of authors,sample size,or first author’s country.Ten of the 35 SPIRIT-AI items(funding,participant timeline,allocation concealment mechanism,implementation,data management,auditing,declaration of interests,access to data,informed consent materials and biological specimens)were not reported by any of the three protocols.Conclusions The reporting and methodological quality of RCTs involving AI need to be improved.Because of the limited availability of protocols,their quality could not be fully judged.Following the CONSORT-AI and SPIRIT-AI statements and with appropriate guidance on the risk of bias when designing and reporting AI-related RCTs can promote standardization and transparency.