The need to develop tailored tools for improving the quality of thematic bibliometric analyses: Evidence from papers published in Sustainability and Scientometrics

Purpose: The aim of this article is to explore up to seven parameters related to the methodological quality and reproducibility of thematic bibliometric research published in the two most productive journals in bibliometrics, Sustainability(a journal outside the discipline) and Scientometrics, the flagship journal in the field.Design/methodology/approach: The study identifies the need for developing tailored tools for improving the quality of thematic bibliometric analyses, and presents a framework that can guide the development of such tools. A total of 508 papers are analysed, 77% of Sustainability, and 23% published in Scientometrics, for the 2019-2021 period.Findings: An average of 2.6 shortcomings per paper was found for the whole sample, with an almost identical number of flaws in both journals. Sustainability has more flaws than Scientometrics in four of the seven parameters studied, while Scientometrics has more shortcomings in the remaining three variables.Research limitations: The first limitation of this work is that it is a study of two scientific journals, so the results cannot be directly extrapolated to the set of thematic bibliometric analyses published in journals from all fields.Practical implications: We propose the adoption of protocols, guidelines, and other similar tools, adapted to bibliometric practice, which could increase the thoroughness, transparency, and reproducibility of this type of research.Originality/value: These results show considerable room for improvement in terms of the adequate use and breakdown of methodological procedures in thematic bibliometric research, both in journals in the Information Science area and journals outside the discipline.

Methodological quality(risk of bias) assessment tools for primary and secondary medical studies: What are they and which is better?

Methodological quality(risk of bias)assessment is an important step before study initiation usage.Therefore,accurately judging study type is the first priority,and the choosing proper tool is also important.In this review,we introduced methodological quality assessment tools for randomized controlled trial(including individual and cluster),animal study,non-randomized interventional studies(including follow-up study,controlled before-and-after study,before-after/pre-post study,uncontrolled longitudinal study,interrupted time series study),cohort study,case-control study,cross-sectional study(including analytical and descriptive),observational case series and case reports,comparative effectiveness research,diagnostic study,health economic evaluation,prediction study(including predictor finding study,prediction model impact study,prognostic prediction model study),qualitative study,outcome measurement instruments(including patient-reported outcome measure development,content validity,structural validity,internal consistency,cross-cultural validity/measurement invariance,reliability,measurement error,criterion validity,hypotheses testing for construct validity,and responsiveness),systematic review and meta-analysis,and clinical practice guideline.The readers of our review can distinguish the types of medical studies and choose appropriate tools.In one word,comprehensively mastering relevant knowledge and implementing more practices are basic requirements for correctly assessing the methodological quality.

Delphi methodology in healthcare research:How to decide its appropriateness

The Delphi technique is a systematic process of forecasting using the collective opinion of panel members.The structured method of developing consensus among panel members using Delphi methodology has gained acceptance in diverse fields of medicine.The Delphi methods assumed a pivotal role in the last few decades to develop best practice guidance using collective intelligence where research is limited,ethically/logistically difficult or evidence is conflicting.However,the attempts to assess the quality standard of Delphi studies have reported significant variance,and details of the process followed are usually unclear.We recommend systematic quality tools for evaluation of Delphi methodology;identification of problem area of research,selection of panel,anonymity of panelists,controlled feedback,iterative Delphi rounds,consensus criteria,analysis of consensus,closing criteria,and stability of the results.Based on these nine qualitative evaluation points,we assessed the quality of Delphi studies in the medical field related to coronavirus disease 2019.There was inconsistency in reporting vital elements of Delphi methods such as identification of panel members,defining consensus,closing criteria for rounds,and presenting the results.We propose our evaluation points for researchers,medical journal editorial boards,and reviewers to evaluate the quality of the Delphi methods in healthcare research.

Methodologic quality of systematic reviews and meta-analyses on the Mediterranean diet and cognitive function： a review

Background and aim： Several systematic reviews and meta-analyses published within the past years have examined the associations of Mediterranean-style diets （MedSDs） on cognitive. However, these reviews have not been evaluated for satisfying Contemporary methodologies quality standards. At the same time, due to the high demand for related care during the aging process in China, it is necessary to summarize the ex-isting system evaluation and evaluate the quality of it, so as to provide strong evidence for the development of relevant work and to standardize the research process and methods in this field. This study evaluated the quality of recent systematic reviews and meta-analyses on MedSD and cognitive outcomes by using an established methodology quality scale, and to provide some valuable references for future research and Chinese systematic review and meta-analysis developing. Methods： We used the AMSTAR Methodology Quality Assessment Scale to evaluate the quality of methodology of the existing meta-analyses and systematic review. Meta analysis and systematic review that meets our criteria must contain the following characteristics： 1） used systematic or meta-analytic procedures to review the lit-erature, and 2） had MedSD interventions independently. Results： Reviews satisfied with 5 to 9 of the AMSTAR MedSD items （mean 7.35, standard deviation： 1.27）, quality is above average with those published in higher-impact journals having a greater quality score. Conclusion： These data suggest that current meta-analyses/systematic reviews evaluating the effect of MedSD on cognitive do not fully meet contemporary methodology quality standards. The methodological quality of the literature needs to be im- proved. As a result, there are more research questions to answer to enhance our understanding of how MedSD affects cognitive. To clarify associations between MedSD and cognitive, future meta-analyses and systematic reviews should not only follow methodology quality standards but also include more statistical modeling results when data allow. And in order to make the conclusions more reliable, we expect higher quality RCTs in the future.

Methodological quality assessment of meta-analyses of cognitive interventions among Alzheimer's disease

Aims:The current study is designed to assess the methodological quality of the meta-analyses of cognitive interventions among Alzheimer's disease,and to investigate the compliance with 16 AMSTAR 2 items.Method:We searched Web of Scie nce,Sino med,PubMed,Embase,and Cochra ne Library from 2016 to 2021,to get the meta-a nalyses of cog nitive in terve nti ons in Alzheimer's disease.The AMSTAR 2 was used to assess the methodological quality.Furthermore,we also explored the complia nce with AMSTAR.Resul ts:We in cluded 9 studies in our research.Of them,6 articles were rated as"extremely low",2 articles as"low"and 1 article as"high".Furthermore,the reporting rates for 16 AMSTAR 2 items ranged from 22.22%to 100%.Conclusion:The methodological quality of meta-analyses of cognitive interventions in Alzheimer's disease is not ideal,and there is still room for improvement.Future studies are supposed to explore the releva nt factors that possibly in flue nce the methodological quality.

Methodological quality evaluation of systematic reviews of music therapy for Alzheimer’s disease in the recent five years

Aims:This study aims to investigate the methodological quality of the systematic reviews of music therapy for Alzheimer’s Disease in the past five years,and to explore their compliance with each AMSTAR(A Measure Tool to Assess Systematic Reviews)item.Based on the above,this study is intended to facilitate the evidence transformation of music therapy in Alzheimer's Disease.Method:Cochrane Library,Web of Science,Embase and PubMed were searched from 2017-2021,to obtain the systematic reviews of music therapy among Alzheimer’s Disease.We used the AMSTAR,to evaluate their methodological quality.Additionally,their compliance with 16 AMSTAR items was investigated.Results:12 systematic reviews were included in this study.The methodological quality of 10 articles was“very low”,1 article was“low”,and 1 article was“high”.The 12 systematic reviews had 25%to 100%compliance with 16 AMSTAR items.Conclusion:The methodological quality of systematic reviews of music therapy among Alzheimer’s Disease in the recent five years is not high,and needs to be further improved.Future research should continue to explore the factors that affect their methodological quality,to promote the transformation of evidence-based evidence.

Appraising the quality of clinical practice guidelines in Chinese patent medicine

Background:Chinese patent medicine(CPM)is a type of traditional Chinese medicine(TCM)that utilizes Chi-nese medicinal materials as its raw ingredients.Following the principles of TCM theory,it undergoes specific processing techniques to create various dosage forms for the prevention and treatment of diseases.The current CPM guidelines,both domestically and internationally,are constrained by limitations in methodological quality and reporting,which do not adequately address the requirements of the majority of clinicians and patients.Objective:To evaluate the methodological rigor and reporting quality of clinical practice guidelines(CPGs)in CPM,with aim to assess their current quality and to provide recommendations for improving guideline develop-ment.Methods:The study utilized the Appraisal of Guidelines for Research and Evaluation II(AGREE II)tool and the Reporting Items for Practice Guidelines in Healthcare for Traditional Chinese Medicine(RIGHT for TCM)checklist to evaluate the methodological rigor and reporting quality of CPM guidelines.After the training session,the four investigators were divided into two groups to evaluate the guidelines of patent Chinese medicine included in this study based on the aforementioned criteria.In case of any discrepancies,a secondary analysis was conducted by all four researchers under the guidance of two clinical experts,ensuring an impartial evaluation outcome.Average score or average reporting rate were calculated for each domain.Results:A total of 32 CPM guidelines that met the inclusion criteria were analyzed.The AGREE II yielded an average score of 67.2%.The domain scores were as follows:scope and purpose(93.3%),stakeholder involve-ment(50.1%),rigor of development(76.8%),clarity of presentation(79.4%),applicability(40.1%),and editorial independence(63.6%).The average reporting rate of the RIGHT for TCM checklist was 70.0%,with the report-ing rates for the seven domains as follows:basic information(94.2%),background(87.9%),evidence(97.5%),recommendation(57.5%),review and quality assurance(11.4%),funding and declaration and management of interests(26.6%),and other information(68.8%).Conclusions:The methodological and reporting quality of CPGs for CPM have been deemed acceptable since 2020.However,there is still room for improvement,particularly in enhancing the reporting quality to provide clinicians and patients with more effective and comprehensive guidance.Additionally,it may be beneficial to develop a comprehensive reporting checklist specifically tailored for CPM guideline development.

Unsatisfied methodological qualities assessment of systematic reviews/Meta-analyses on Chinese medicine for stroke and their risk factors

Background:Stroke is not only high in morbidity and mortality but also poses a great burden of disease and it is also the most reported disease in Chinese medicine systematic reviews.Therefore,the quality of such evidence couldn’t be ignored.This study aims to use a measurement tool to assess systematic reviews(AMSTAR)to assess the methodological qualities of SR/Meta-analyses of Chinese medicine on stroke.Methods:Systematic searching of seven electronic databases and PROSPERO registration platform was carried out.Two researchers separately selected studies,extracted bibliographical characteristics and scored every included study independently after training.Total score and the proportion of each item completion were explored in different subgroup comparisons.Spearman rank correlation and multivariable logistic regression were used to measure the association between bibliographical characteristics and total score or each item.Results:Total average score of AMSTAR 1.0 checklists of 234 systematic reviews/Meta-analyses was 4.47(95%CI 4.27–4.68)and the qualities were unsatisfied especially in terms of priori setting(2.14%),grey literature inclusion(5.13%),providing a list of excluded studies(2.14%)and conflict of interest(0.00%).No improvement was found in 3 years even after the publication of AMSTAR.Chinese or nonregistered systematic reviews/Meta-analyses showed even worse methodological qualities(P<0.01).Positive correlation was found between individual items and number of pages,number of authors,research questions,languages or Meta-analyse separately(P<0.05).Conclusion:The methodological qualities of systematic reviews/Meta-analyses of Chinese medicine on stroke are poor especially Chinese studies,non-registered studies,brief studies and studies without Meta-analyse or cooperation.There is no obvious improvement over these years even after the publication of AMSTAR tool,so it is urgent to promote the use of AMSTAR or develop other efficient methods to control the quantity and monitor the quality in future.

Clinical practice guidelines for traditional Chinese medicine and integrated traditional Chinese and western medicine:a cross-sectional study of data analysis from 2010 to 2020

Objective With the increasing publication of clinical practice guidelines(CPG)for Traditional Chinese Medicine(TCM)and Integrated Traditional Chinese and Western Medicine(IM),the standardization and scientifiction of its formulation have gradually attracted many people’s attention.To offer an overview of TCM and IM CPGs published over the past decade and analyze their general characteristics and methodological quality.Methods The China National Knowledge Infrastructure(CNKI)and WANFANG databases were searched for clinical practice guidelines and expert consensus papers from January 2010 to June 2021.Two researchers independently completed the literature screening and cross-checking according to the inclusion and exclusion criteria of CPGs and extracted information on general characteristics and methodological quality of CPGs.Results According to the selection criteria,231 CPGs(EB-CPGs=119,CBCPGs=112)were selected and the number of CPGs published in the 11 years showed an overall upward trend.The vast majority of CPGs used the Western naming system for the diseases,and only 11 CPGs were named of TCM diseases or symptoms.TCM treatments were recommended in 223 CPGs.There were 156 ancient Chinese Medicine literature sources cited in 231 CPGs and opinions and experiences of 62 TCM experts cited in 37 CPGs.The methodological quality of EB-CPGs for TCM and IM were significantly better than CB-CPGs in 11 items.Only 60 EB-CPGs and 7 CB-CPGs designated clear criteria for grading quality of evidence and strength of the recommendations and 74 CPGs presented both the level of evidence and the strength of recommendations.We classified all CPGs according to whether or not they used GRADE,and the results showed that the CPGs using GRADE had higher methodological quality and more standardized reports.Conclusion This research has shown that the quantity and quality of CPGs in both TCM and IM have improved over the time span,but the methodological quality,especially evidence citation,and the use of criteria for grading quality of evidence and strength of the recommendations,still needs to further improvement in the future.

Methological Quality of Systematic Reviews and Meta-Analyses on Acupuncture for Stroke: A Review of Review

Objective： To assess the methodological quality of systematic reviews and meta-analyses regarding acupuncture intervention for stroke and the primary studies within them. Methods： Two researchers searched Pub Med, Cumulative index to Nursing and Allied Health Literature, Embase, ISI Web of Knowledge, Cochrane, Allied and Complementary Medicine, Ovid Medline, Chinese Biomedical Literature Database, China National Knowledge Infrastructure, Wanfang and Traditional Chinese Medical Database to identify systematic reviews and meta-analyses about acupuncture for stroke published from the inception to December 2016. Review characteristics and the criteria for assessing the primary studies within reviews were extracted. The methodological quality of the reviews was assessed using adapted Oxman and Guyatt Scale. The methodological quality of primary studies was also assessed. Results： Thirty-two eligible reviews were identified, 15 in English and 17 in Chinese. The English reviews were scored higher than the Chinese reviews（P=0.025）, especially in criteria for avoiding bias and the scope of search. All reviews used the quality criteria to evaluate the methodological quality of primary studies, but some criteria were not comprehensive. The primary studies, in particular the Chinese reviews, had problems with randomization, allocation concealment, blinding, dropouts and withdrawals, intent-to-treat analysis and adverse events. Conclusions： Important methodological flaws were found in Chinese systematic reviews and primary studies. It was necessary to improve the methodological quality and reporting quality of both the systematic reviews published in China and primary studies on acupuncture for stroke.

Investigation and evaluation of randomized controlled trials for interventions involving artificial intelligence

Objective Complete and transparent reporting is of critical importance for randomized controlled trials(RCTs).The present study aimed to determine the reporting quality and methodological quality of RCTs for interventions involving artificial intelligence(AI)and their protocols.Methods We searched MEDLINE(via PubMed),Embase,Web of Science,CBMdisc,Wanfang Data,and CNKI from January 1,2016,to November 11,2020,to collect RCTs involving AI.We also extracted the protocol of each included RCT if it could be obtained.CONSORT-AI(Consolidated Standards of Reporting Trials-Artificial Intelligence)statement and Cochrane Collaboration’s tool for assessing risk of bias(ROB)were used to evaluate the reporting quality and methodological quality,respectively,and SPIRIT-AI(The Standard Protocol Items:Recommendations for Interventional Trials-Artificial Intelligence)statement was used to evaluate the reporting quality of the protocols.The associations of the reporting rate of CONSORT-AI with the publication year,journal’s impact factor(IF),number of authors,sample size,and first author’s country were analyzed univariately using Pearson’s chi-squared test,or Fisher’s exact test if the expected values in any of the cells were below 5.The compliance of the retrieved protocols to SPIRIT-AI was presented descriptively.Results Overall,29 RCTs and three protocols were considered eligible.The CONSORT-AI items“title and abstract”and“interpretation of results”were reported by all RCTs,with the items with the lowest reporting rates being“funding”(0),“implementation”(3.5%),and“harms”(3.5%).The risk of bias was high in 13(44.8%)RCTs and not clear in 15(51.7%)RCTs.Only one RCT(3.5%)had a low risk of bias.The compliance was not significantly different in terms of the publication year,journal’s IF,number of authors,sample size,or first author’s country.Ten of the 35 SPIRIT-AI items(funding,participant timeline,allocation concealment mechanism,implementation,data management,auditing,declaration of interests,access to data,informed consent materials and biological specimens)were not reported by any of the three protocols.Conclusions The reporting and methodological quality of RCTs involving AI need to be improved.Because of the limited availability of protocols,their quality could not be fully judged.Following the CONSORT-AI and SPIRIT-AI statements and with appropriate guidance on the risk of bias when designing and reporting AI-related RCTs can promote standardization and transparency.