Development of the Documentation and Appraisal Review Tool for systematic reviews

AIM:To develop a tool to more explicitly assess and document the quality of systematic reviews.METHODS:We developed the Documentation and Appraisal Review Tool(DART)using epidemiologic principles of study design and the following resources:the modified Overview Quality Assessment Questionnaire(modified OQAQ),Assessment of Multiple Systematic Reviews(AMSTAR),the Cochrane Handbook,and the standards promoted by the Agency for Healthcare Research and Quality,and the Institutes of Medicine(IOM).We designed the DART tool to include the following:more detail to provide guidance and improve standardization of use,an approach to assess quality of systematic reviews addressing a variety of research designs,and additional space for recording notes to facilitate recall.DART underwent multiple rounds of testing with methodologists of varying levels of training and experience.Based on the results of six phases of pilot testing,we revised DART to improve performance,clarity and consistency.Pilot testing also included comparisons between DART,and the two most commonly used tools to evaluate the quality of systematic reviews,the modified OQAQ and AMSTAR.RESULTS:Compared to AMSTAR and modified OQAQ,DART includes two unique questions and several questions covered by modified OQAQ or AMSTAR but not both.Modified OQAQ and DART had the highest reporting consistency.Four AMSTAR questions were unclear and elicited inconsistent responses.Identifying reviewer rationale was most difficult using the modified OQAQ tool,and easiest using DART.DART allowsfor documentation of reviewer rationale,facilitating reconciliation between reviewers and documentation for future updates.DART also provides a comprehensive,systematic approach for reviewers with limited experience with systematic review methodology,to critically analyze systematic reviews.In addition,DART is the only one of the three tools to explicitly include quality review for biases specific to observational studies.This is now more widely recognized as important for assessing risk in order to generate recommendations that balance benefit to harm.The tool also includes the assessment of standards recommended by the March 2011 IOM Standards for Systematic Review.CONCLUSION:This comprehensive tool improves upon existing tools for assessing the quality of systematic reviews and guides reviewers through critically analyzing a systematic review.