1	Systematic Literature Reviews and Analytic Frameworks David Atkins, MD, MPH Center for Outcomes and Evidence Agency for Healthcare Research and Quality
2	Issues to Consider How can we do a better job determining when and what large trials are worth pursuing? When is evidence sufficient for clinical decision-making? When evidence is not sufficient, what types of studies are needed?
3	Rationale for Systematic Reviews AIM: To reduce potential for bias and error Explosion of medical literature – 400,000 Medline/yr Complex links between interventions and outcomes Effects may be modest, develop over long time Need to balance benefits, harms and costs Variable design and quality of published studies Internal validity External validity (generalizability or applicability)
4	Attributes of Systematic Reviews Explicit methods Complete and unbiased review Transparent reasoning Consistent (reproducible) results AIMS: Facilitate decision making Identify critical research gaps
5	Steps in Systematic Review Process Clarify critical questions: Analytic framework Complete and unbiased search for evidence Assess quality of evidence Link conclusions to: -- quality of evidence -- balance of benefits vs. harms
6	Analytic Framework - 1
7	2. Unbiased search for evidence What outcomes are relevant? Intermediate measures vs. health outcomes What types of studies are included/excluded? Animal vs. human? Developing vs. developed countries? What study designs are adequate? RCTs vs. observational studies
8	3. Assess quality of evidence What do we mean by quality? “Extent to which a study’s design, conduct, and analysis has minimized selection, measurement, and confounding biases.” Lohr, J Qual Improvement, 1999 “Extent to which one can be confident that an estimate of effect is correct” GRADE , BMJ 2004
9	Quality of Individual Studies GOAL: Identify those studies with highest quality (internal validity) Depends on study design (e.g., RCT vs. cohort) Depends on study execution (e.g., blinding) Critical elements vary by study design and specific topic Best established for RCTs Evolving methods for observational studies
10	3 levels of quality Quality of individual studies Overall quality of studies addressing a specific question Quality of collected evidence to inform a clinical issue
11	Analytic Framework - 1
12	4. Synthesis and Reporting Summarize evidence for each key question Strength of evidence depends on internal validity, external validity, and other factors Quantitative or qualitative synthesis as appropriate Conclusions linked to evidence in explicit and transparent way
13	Analytic Framework – Use for clinical decisions
14	Analytic Framework – Use research
15	Challenges in Systematic Reviews Labor intensive Not necessarily more than any comprehensive review Requires more up-front work Specify questions, outcomes, designs May find little high quality literature or be overwhelmed by volume of literature Expand or restrict scope if needed
16	Other considerations Systematic reviews will not eliminate controversy over evidence or resolve contentious policy questions Conflicts over evidence or policy usually reflect conflicts over values What constitutes “good enough” evidence? What outcomes are most important? When is a benefit “important”?
17	Conclusions Systematic reviews provide more consistent and transparent approach to summarizing existing evidence Useful for developing clinical recommendations Useful for identifying research gaps or important implications of policy decisions May separate questions of evidence from questions of values