<< Back to Agenda

Slide 1: Systematic Literature Reviews and Analytic Frameworks

David Atkins, MD, MPH
Center for Outcomes and Evidence
Agency for Healthcare Research and Quality

Slide 2: Issues to Consider

• How can we do a better job determining when and what large trials are worth pursuing?

• When is evidence sufficient for clinical decision-making?
• When evidence is not sufficient, what types of studies are needed?

Slide 3: Rationale for Systematic Reviews

• AIM: To reduce potential for bias and error
• Explosion of medical literature – 400,000 Medline/yr
• Complex links between interventions and outcomes
• Effects may be modest, develop over long time
• Need to balance benefits, harms and costs
• Variable design and quality of published studies
  • Internal validity
  • External validity ( generalizability or applicability)

Slide 4: Attributes of Systematic Reviews

• Explicit methods
• Complete and unbiased review
• Transparent reasoning
• Consistent (reproducible) results

AIMS:

• Facilitate decision making
• Identify critical research gaps

Slide 5: Steps in Systematic Review Process

1. Clarify critical questions: Analytic framework
2. Complete and unbiased search for evidence
3. Assess quality of evidence
4. Link conclusions to:

-- quality of evidence
-- balance of benefits vs. harms

Slide 6: Analytic Framework - 1

Slide 7: 2. Unbiased search for evidence

• What outcomes are relevant?
• Intermediate measures vs. health outcomes
• What types of studies are included/excluded?
• Animal vs. human?
• Developing vs. developed countries?
• What study designs are adequate?
• RCTs vs. observational studies

Slide 8: 3. Assess quality of evidence

• What do we mean by quality?

“Extent to which a study’s design, conduct, and analysis has minimized selection, measurement, and confounding biases.”

Lohr,   J Qual Improvement, 1999

“Extent to which one can be confident that an estimate of effect is correct”

GRADE , BMJ 2004

Slide 9: Quality of Individual Studies

• GOAL: Identify those studies with highest quality (internal validity)
• Depends on study design (e.g., RCT vs. cohort)
• Depends on study execution (e.g., blinding)
• Critical elements vary by study design and specific topic
  • Best established for RCTs
  • Evolving methods for observational studies

Slide 10: 3 levels of quality

• Quality of individual studies
• Overall quality of studies addressing a specific question
• Quality of collected evidence to inform a clinical issue

Slide 11: Analytic Framework - 1

Slide 12: 4. Synthesis and Reporting

• Summarize evidence for each key question
  • Strength of evidence depends on internal validity,   external validity, and other factors  
• Quantitative or qualitative synthesis as appropriate
• Conclusions linked to evidence in explicit and transparent way

Slide 13: Analytic Framework – Use for clinical decisions

Slide 14: Analytic Framework – Use research

Slide 15: Challenges in Systematic Reviews

• Labor intensive
  • Not necessarily more than any comprehensive review
• Requires more up-front work
  • Specify questions, outcomes, designs
• May find little high quality literature or be overwhelmed by volume of literature
  • Expand or restrict scope if needed

Slide 16: Other considerations

• Systematic reviews will not eliminate controversy over evidence or resolve contentious policy questions
• Conflicts over evidence or policy usually reflect conflicts over values
  • What constitutes “good enough” evidence?
  • What outcomes are most important?
  • When is a benefit “important”?

Slide 17: Conclusions

• Systematic reviews provide more consistent and transparent approach to summarizing existing evidence
• Useful for developing clinical recommendations
• Useful for identifying research gaps or important implications of policy decisions
• May separate questions of evidence from questions of values

<< Back to Agenda