<< Back to Agenda

Slide 1: Distinguishing Causal from Noncausal Associations: Issues Intrinsic to Behavioral and Other Complex Interventions

William R. Shadish
University of California, Merced

Slide 2: Photograph of UC Merced construction November 12, 2004

Slide 3: Overview of This Talk

• In general, I'm going to discuss the logical, design, and statistical tools that we use to make causal inferences, and point to problems in each of those areas that cause us to make inferential errors
• Most of the issues are not specific to complex or behavioral interventions, though I will point out some particular problems in that area.

Slide 4: Three traditional tools to help judge treatment effectiveness

• Logic
• Design
• Statistics

Slide 5: Logic

• ausal inference is always, first and foremost, a matter of logic.
• Mill's Canons (temporal precedence, covariation, alternatives)
• Counterfactual Models (e.g., Rubin's model)
• This means causal inference is possible with nonrandomized designs
• Smoking and cancer in humans
• Dietary treatment of PKU (short time series)
• Campbell's (1975) Degrees of Freedom and the Case Study
• But the conditions facilitating causal inference in case studies are rare

Slide 6: Example of Aids to Logic: Campbell's Validity Types

• Statistical Conclusion Validity (did the treatment covary with the outcome?)
• Internal Validity (did the treatment affect the outcome?)
• Construct Validity (what labels or constructs best represent what we did?)
• External Validity (to what does the effect generalize?)

Slide 7: Threats to Validity

• Threats to Internal Validity
• Ambiguous Causal Precedence
• History
• Maturation
• Attrition
• Selection
• Regression
• Testing
• Instrumentation
• The idea is to rule each of these out as alternative explanations for the observed effect
• Similar lists of threats to the other validities
• Similar lists in other literatures, such as Sackett (1979) for case control studies

Slide 8: Logical Problems

• Many scientists are surprisingly unaware of basic causal logic
• Our ability to enumerate specific contextual threats to validity is limited.
• Need research programs to identify threats
• Need ways to quantify qualitative threats to validity
• Scientists are just people, and people are not good at making causal inferences based on logic alone.
• E.g., Confirmation bias.

Slide 9: Design

• Randomized experiments (lots of examples of their feasibility with behavioral interventions, though some special problems do arise)
• Strong quasi-experiments
• Interrupted time series (on groups or individuals)
• Regression discontinuity (cutoff assignment)
• Weaker quasi-experiments
• Nonequivalent comparison group designs
• Observational Studies
• Meta-Analysis
• Of randomized experiments
• Of mixed methods work

Slide 10: Design Problems

• Specious arguments against randomization
• Occasional premature randomized experiments
• Misplaced faith in one large randomized trial over many small randomized trials. Both are important, but:
• Many small trials allows exploration of generalization
• Under-use of stronger quasi-experiments
• Failure to take full advantage of meta-analysis to explore generalization of effects
• Response surface modeling using multimethod databases
• Routine exploration of generalization over moderators
• Misplaced faith in single meta-analyses
• They are only as good as the data going into them
• E.g., most negative effects for Vitamin E occurred in studies using both Vit E and beta-carotene

-

Slide 11: Statistical Tools

• In Randomized Experiments
• Attrition Analyses
• Incomplete treatment implementation
• Analysis of nested data
• In Quasi-Experiments
• Selection Bias Modeling
• Propensity Score and Hidden Bias Analysis
• In Meta-Analysis
• Response Surface Modeling

Slide 12: Statistical Problems

• Lack of awareness of recent developments for both randomized and nonrandomized experiments.
• E.g., continued failure to take nesting into account, leading to Type I errors
• Misplaced preference for single point answers rather than bracketing
• Lack of clear methods or software to do some of the analyses (e.g., response surface modeling in meta-analysis)

Slide 13: My Favorite Top 3 Problems

1. Misplaced faith in "singletons"

• Single point estimates
• Single studies
• Single meta-analyses

1. Misanalyzed data (especially nesting and attrition)
2. Failure to take generalization as seriously as causation in design and analysis

Slide 14: But we should not overstate the problem

• We do want to minimize preventable problems, but:
• Experiments are about DISCOVERY of effects of treatments. We should not be upset that the effects are not always what we anticipated.
• We rarely know when the "last word" is in (even for HRT or beta-carotene).

<< Back to Agenda