Analysis of COVID-19 Guideline Quality and Change of Recommendations:A Systematic Review

Background.Hundreds of coronavirus disease 2019(COVID-19)clinical practice guidelines(CPGs)and expert consensus statements have been developed and published since the outbreak of the epidemic.However,these CPGs are of widely variable quality.So,this review is aimed at systematically evaluating the methodological and reporting qualities of COVID-19 CPGs,exploring factors that may influence their quality,and analyzing the change of recommendations in CPGs with evidence published.Methods.We searched five electronic databases and five websites from 1 January to 31 December 2020 to retrieve all COVID-19 CPGs.The assessment of the methodological and reporting qualities of CPGs was performed using the AGREE II instrument and RIGHT checklist.Recommendations and evidence used to make recommendations in the CPGs regarding some treatments for COVID-19(remdesivir,glucocorticoids,hydroxychloroquine/chloroquine,interferon,and lopinavir-ritonavir)were also systematically assessed.And the statistical inference was performed to identify factors associated with the quality of CPGs.Results.We included a total of 92 COVID-19 CPGs developed by 19 countries.Overall,the RIGHT checklist reporting rate of COVID-19 CPGs was 33.0%,and the AGREE II domain score was 30.4%.The overall methodological and reporting qualities of COVID-19 CPGs gradually improved during the year 2020.Factors associated with high methodological and reporting qualities included the evidence-based development process,management of conflicts of interest,and use of established rating systems to assess the quality of evidence and strength of recommendations.The recommendations of only seven(7.6%)CPGs were informed by a systematic review of evidence,and these seven CPGs have relatively high methodological and reporting qualities,in which six of them fully meet the Institute of Medicine(IOM)criteria of guidelines.Besides,a rapid advice CPG developed by the World Health Organization(WHO)of the seven CPGs got the highest overall scores in methodological(72.8%)and reporting qualities(83.8%).Many CPGs covered the same clinical questions(it refers to the clinical questions on the effectiveness of treatments of remdesivir,glucocorticoids,hydroxychloroquine/chloroquine,interferon,and lopinavirritonavir in COVID-19 patients)and were published by different countries or organizations.Although randomized controlled trials and systematic reviews on the effectiveness of treatments of remdesivir,glucocorticoids,hydroxychloroquine/chloroquine,interferon,and lopinavir-ritonavir for patients with COVID-19 have been published,the recommendations on those treatments still varied greatly across COVID-19 CPGs published in different countries or regions,which may suggest that the CPGs do not make sufficient use of the latest evidence.Conclusions.Both the methodological and reporting qualities of COVID-19 CPGs increased over time,but there is still room for further improvement.The lack of effective use of available evidence and management of conflicts of interest were the main reasons for the low quality of the CPGs.The use of formal rating systems for the quality of evidence and strength of recommendations may help to improve the quality of CPGs in the context of the COVID-19 pandemic.During the pandemic,we suggest developing a living guideline of which recommendations are supported by a systematic review for it can facilitate the timely translation of the latest research findings to clinical practice.We also suggest that CPG developers should register the guidelines in a registration platform at the beginning for it can reduce duplication development of guidelines on the same clinical question,increase the transparency of the development process,and promote cooperation among guideline developers all over the world.Since the International Practice Guideline Registry Platform has been created,developers could register guidelines prospectively and internationally on this platform.

Investigation and evaluation of randomized controlled trials for interventions involving artificial intelligence

Objective Complete and transparent reporting is of critical importance for randomized controlled trials(RCTs).The present study aimed to determine the reporting quality and methodological quality of RCTs for interventions involving artificial intelligence(AI)and their protocols.Methods We searched MEDLINE(via PubMed),Embase,Web of Science,CBMdisc,Wanfang Data,and CNKI from January 1,2016,to November 11,2020,to collect RCTs involving AI.We also extracted the protocol of each included RCT if it could be obtained.CONSORT-AI(Consolidated Standards of Reporting Trials-Artificial Intelligence)statement and Cochrane Collaboration’s tool for assessing risk of bias(ROB)were used to evaluate the reporting quality and methodological quality,respectively,and SPIRIT-AI(The Standard Protocol Items:Recommendations for Interventional Trials-Artificial Intelligence)statement was used to evaluate the reporting quality of the protocols.The associations of the reporting rate of CONSORT-AI with the publication year,journal’s impact factor(IF),number of authors,sample size,and first author’s country were analyzed univariately using Pearson’s chi-squared test,or Fisher’s exact test if the expected values in any of the cells were below 5.The compliance of the retrieved protocols to SPIRIT-AI was presented descriptively.Results Overall,29 RCTs and three protocols were considered eligible.The CONSORT-AI items“title and abstract”and“interpretation of results”were reported by all RCTs,with the items with the lowest reporting rates being“funding”(0),“implementation”(3.5%),and“harms”(3.5%).The risk of bias was high in 13(44.8%)RCTs and not clear in 15(51.7%)RCTs.Only one RCT(3.5%)had a low risk of bias.The compliance was not significantly different in terms of the publication year,journal’s IF,number of authors,sample size,or first author’s country.Ten of the 35 SPIRIT-AI items(funding,participant timeline,allocation concealment mechanism,implementation,data management,auditing,declaration of interests,access to data,informed consent materials and biological specimens)were not reported by any of the three protocols.Conclusions The reporting and methodological quality of RCTs involving AI need to be improved.Because of the limited availability of protocols,their quality could not be fully judged.Following the CONSORT-AI and SPIRIT-AI statements and with appropriate guidance on the risk of bias when designing and reporting AI-related RCTs can promote standardization and transparency.