1	Why Do We Sometimes Get It Wrong? What can we learn from formal systems of assessing the evidence and making recommendations? Comments based primarily on process of USPSTF
2	Two Types of Errors “Unfortunate adoption”: widespread adoption of interventions later found to have net harm. “Unfortunate delay”: delayed adoption of interventions later found to have net benefit.
3	3 Steps to Getting It Right Step 1: Assess the evidence (benefits and harms) without bias Step 2: Make a general policy recommendation Step 3: Decide about action (or inaction)
4	USPSTF Overview Step 1: Assess Evidence Carefully define the key question(s) and the criteria for admissible evidence Define outcomes: usually skeptical of intermediate outcomes (even RCTs) Admissible evidence: prefers but does not require RCTs; reviews other evidence when needed
5	Which Evidence? Other Useful Designs Non-randomized studies with concurrent, well-matched controls Longer-term cohort studies of natural history with/without treatment Population-based prevalence (cross-sectional) studies Diagnostic accuracy studies Studies of psychological and physical harms
6	USPSTF Overview Step 1: Assess Evidence Systematically review the evidence for the key questions: Critical appraisal of internal validity Synthesize results for each question Consider extrapolation within the source population Consider external validity of results
7	USPSTF Overview Step 1: Assess Evidence In its consideration of the evidence, the USPSTF uses no rigid formula. Instead it relies on judgment based on explicit criteria. It seeks to be transparent about its rationale for its judgments.
8	USPSTF Overview Step 1: Assess Evidence Synthesizing evidence: unexplained inconsistency is taken seriously. Biologic plausibility is a contributory factor; it does not automatically determine the conclusion. The issue is not simply whether benefits exist; it is whether we can estimate the magnitude of any benefits.
9	USPSTF Overview Step 1: Assess Evidence Equal emphasis is placed on searching for evidence of harms as for benefits. Harms are usually present but often under-researched. No evidence of harm is not interpreted as evidence of no harm.
10	USPSTF Overview Step 1: Assess Evidence Searching for harms often requires appraising different types of research design from benefits. Still requires studies of reasonable quality and consistency.
11	Assessing Evidence: Judgment is Required Should acknowledge that judgment is required at several steps in assessing evidence. Defining admissible evidence Critical appraisal of internal validity Synthesis of internally valid evidence Willingness to extrapolate Critical appraisal of external validity
12	Judgment, Opinion, or Necessary Flexibility? Explicitly label statements of judgment Base judgments on a priori criteria Give rationale for judgment
13	USPSTF Overview Step 2: Recommend a Policy Determine whether the evidence is sufficient to estimate the magnitude of benefits and harms. If so, weigh the magnitude and value of benefits against the magnitude and value of harms on a population level; base recommendation on net benefits.
14
15	Beyond the USPSTF Step 3: Make a Decision After the recommendation, decision makers must decide about implementation. Local factors are involved: Costs/resources Availability of intervention (trained personnel; technology; quality) Competing priorities of health care system or patients
16	A Problem We don’t have complete evidence about everything. Sometimes it makes sense to implement an intervention before we have adequate information to judge its consequences. In these cases, decision makers may become confused and assume that interventions that they are actively implementing are backed by better evidence than is the case.
17	Two Wise Men ▪ Mark Twain: “It ain’t what people don’t know that hurts them. It’s what they know that ain’t so.” ▪ Stephen Hawking: “The greatest enemy of knowledge is not ignorance; it is the illusion of knowledge.”
18	Overall Summary The USPSTF process is not rigid or formulaic. It allows for judgment and extrapolation. It admittedly errs on the skeptical side. It is not hesitant to admit uncertainty; that we don’t know the answer.
19	My Judgment (Possibly Biased) Such a process will probably: Occasionally, but not often, lead to unfortunate adoption Occasionally, but not often, lead to unfortunate delay.
20	Overall Summary The art in the science is getting the right balance between unfortunate adoption and unfortunate delay. The optimal place for that balance is a value judgment.