<< Back to Agenda

Slide 1: Why Do We Sometimes Get It Wrong?

• What can we learn from formal systems of assessing the evidence and making recommendations?
• Comments based primarily on process of USPSTF

Slide 2: Two Types of Errors

• "Unfortunate adoption": widespread adoption of interventions later found to have net harm.
• "Unfortunate delay": delayed adoption of interventions later found to have net benefit.

Slide 3: 3 Steps to Getting It Right

• Step 1: Assess the evidence (benefits and harms) without bias
• Step 2: Make a general policy recommendation
• Step 3: Decide about action (or inaction)

Slide 4: USPSTF Overview Step 1: Assess Evidence

• Carefully define the key question(s) and the criteria for admissible evidence
• Define outcomes: usually skeptical of intermediate outcomes (even RCTs)
• Admissible evidence: prefers but does not require RCTs; reviews other evidence when needed

Slide 5: Which Evidence? Other Useful Designs

• Non-randomized studies with concurrent, well-matched controls
• Longer-term cohort studies of natural history with/without treatment
• Population-based prevalence (cross-sectional) studies
• Diagnostic accuracy studies
• Studies of psychological and physical harms

Slide 6: USPSTF Overview Step 1: Assess Evidence

• Systematically review the evidence for the key questions:
• Critical appraisal of internal validity
• Synthesize results for each question
• Consider extrapolation within the source population
• Consider external validity of results

Slide 7: USPSTF Overview Step 1: Assess Evidence

• In its consideration of the evidence, the USPSTF uses no rigid formula.
• Instead it relies on judgment based on explicit criteria.
• It seeks to be transparent about its rationale for its judgments.

Slide 8: USPSTF Overview Step 1: Assess Evidence

• Synthesizing evidence: unexplained inconsistency is taken seriously.
• Biologic plausibility is a contributory factor; it does not automatically determine the conclusion.
• The issue is not simply whether benefits exist; it is whether we can estimate the magnitude of any benefits.

Slide 9: USPSTF Overview Step 1: Assess Evidence

• Equal emphasis is placed on searching for evidence of harms as for benefits.
• Harms are usually present but often under-researched.
• No evidence of harm is not interpreted as evidence of no harm.

Slide 10: USPSTF Overview Step 1: Assess Evidence

• Searching for harms often requires appraising different types of research design from benefits.
• Still requires studies of reasonable quality and consistency.

Slide 11: Assessing Evidence: Judgment is Required

• Should acknowledge that judgment is required at several steps in assessing evidence.
• Defining admissible evidence
• Critical appraisal of internal validity
• Synthesis of internally valid evidence
• Willingness to extrapolate
• Critical appraisal of external validity

Slide 12: Judgment, Opinion, or Necessary Flexibility?

• Explicitly label statements of judgment
• Base judgments on a priori criteria
• Give rationale for judgment

Slide 13: USPSTF Overview Step 2: Recommend a Policy

• Determine whether the evidence is sufficient to estimate the magnitude of benefits and harms.
• If so, weigh the magnitude and value of benefits against the magnitude and value of harms on a population level; base recommendation on net benefits.

Slide 14: USPSTF Recommendation Grid (Table)

Slide 15: Beyond the USPSTF Step 3: Make a Decision

• After the recommendation, decision makers must decide about implementation.
• Local factors are involved:
• Costs/resources
• Availability of intervention (trained personnel; technology; quality)
• Competing priorities of health care system or patients

Slide 16: A Problem

• We don't have complete evidence about everything.
• Sometimes it makes sense to implement an intervention before we have adequate information to judge its consequences.
• In these cases, decision makers may become confused and assume that interventions that they are actively implementing are backed by better evidence than is the case.

Slide 17: Two Wise Men

• Mark Twain: "It ain't what people don't know that hurts them. It's what they know that ain't so."
• Stephen Hawking: "The greatest enemy of knowledge is not ignorance; it is the illusion of knowledge."

Slide 18: Overall Summary

• The USPSTF process is not rigid or formulaic. It allows for judgment and extrapolation.
• It admittedly errs on the skeptical side.
• It is not hesitant to admit uncertainty; that we don't know the answer.

Slide 19: My Judgment (Possibly Biased)

• Such a process will probably:
• Occasionally, but not often, lead to unfortunate adoption
• Occasionally, but not often, lead to unfortunate delay.

Slide 20-Overall Summary

• The art in the science is getting the right balance between unfortunate adoption and unfortunate delay.
• The optimal place for that balance is a value judgment.

<< Back to Agenda