1	Moving from Observational Studies to Clinical Trials: Why do We Sometimes Get It Wrong? Joseph Lau, MD Rapporteur
2	Evaluating Study Outcomes: Biomarkers, Intermediate Endpoints, and Surrogate Endpoints Ross Prentice, PhD Surrogate Endpoint Definition and Application Stuart Baker, ScD Recent Approaches to Surrogate Endpoint Validation David Ransohoff, MD New Complexity: The “Omics” Revolution Daniel Hayes, MD Methods of Biomarker Validation
3	Ross Prentice, PhD: Surrogate Endpoint Definition and Application Surrogate outcome definition Conceptual framework for associations of treatment, surrogate, and true endpoint Proposed meta-analysis approach of borrowing information in prior studies of similar treatments in similar populations
4	Stuart Baker, ScD: Recent Approaches to Surrogate Endpoint Validation Process of validating markers or endpoints Hypothesis testing framework Estimation framework Recommended meta-analysis estimation approach to validate surrogate endpoint Real examples? Has this method been validated empirically? Other approaches? Bayesian method?
5	David Ransohoff, MD: New Complexity: The “Omics” Revolution Promises and disappointments of cancer markers Rules of evidence not well developed Current overly optimistic interpretation of “omics” data Bias as threat to validity
6	Daniel Hayes, MD: Methods of Oncology Biomarker Validation Many proposed tumor markers Most inadequately validated
7	A few comments on other sessions Current concepts How best to evaluate existing evidence? How to design better future studies?
8	Issues evaluating evidence: an EBM-er’s perspective Evidence is seldom single sourced (basic science, animal, human observations, human experiments) Observational studies vs RCTs Surrogates vs clinical outcomes Mega-trials vs (meta-analyses) small trials Large RCTs vs large RCTs Methodological quality of the studies Publication bias
9	Comparisons of RCTs with NROS BMJ 1998; Oxman et al. NEJM 2000; Concato et al. NEJM 2000; Benson et al. JAMA 2001; Ioannidis et al.
10	Comparison of RCTs and NROS in meta-analyses Ioannidis et al. JAMA 2001:286:821-830 A total of 45 topics were considered. They were identified from comprehensive searches of MEDLINE, The Cochrane Library, previous relevant publications and personal archives – c. 3,000 meta-analyses were screened. The 45 topics included 408 primary studies with available binary data (240 RCTs and 168 NROS) NROS included 71 prospective studies, 40 retrospective cohort studies, 25 case-control studies, 29 studies with historical controls, and 3 studies with unclear designs
11
12	Comparisons between randomized and non-randomized evidence. Ioannidis J. et al. JAMA 2001;286:821-830.
13	Comparisons between randomized and non-randomized evidence. Ioannidis J. et al. JAMA 2001;286:821-830.
14	Heterogeneity in RCTs and in NROS Ioannidis et al. JAMA 2001;286:821-830. Statistically significant heterogeneity between randomized trials was seen in 9 of 39 topics with at least 2 RCTs included Statistically significant heterogeneity between the non-randomized studies was seen in 13 of 32 topics with at least 2 NROS included The estimated between-study heterogeneity tended to be smaller among RCTs than among NROS (p=0.032)
15	Comparison of the magnitude of treatment effects Ioannidis J. et al. JAMA 2001;286:821-830. In 25 of 45 cases, the non-randomized studies showed a larger treatment effect for the experimental treatment than the randomized trials. The opposite occurred in 14 cases, but it was a data artifact in 3 of them. In 6 topics there was either no clear-cut experimental arm or the effects were similar (p=0.009).
16	Discrepancies between RCTs and NROS Ioannidis J. et al. JAMA 2001;286:821-830. Discrepancies beyond chance were observed in 12 of 45 cases by fixed effects and in 7 of 45 cases by random effects In these discrepancies, almost always the treatment effect was more favorable in NROS When limiting analyses to prospective studies, there were disagreements in 2 of 26 topics (8%)
17	Conclusions Ioannidis J. et al. JAMA 2001;286:821-830. Treatment effects in RCTs and observational studies on the same topic tend to be highly correlated Nevertheless, discrepancies do occur in about 1 out of 6 cases, even when between-study heterogeneity is accounted for Typically, discrepant pairs tend to show more favorable results in observational studies Discrepancies in the absolute magnitude of effect (=“how much it works”) are very common
18	Conclusions (cont) Ioannidis J. et al. JAMA 2001;286:821-830. Observational studies exhibit larger variability in their treatment effects than RCTs Discrepancies are more common when retrospective observational designs are considered Both RCTs and NROS must be carefully scrutinized for sources of genuine heterogeneity and bias RCTs and NROS should not be seen as mutually exclusive domains of research
19	Comparisons of Large RCTs with Meta-analyses of small trials Villar et al. Lancet 1995 Cappelleri et al. JAMA 1996 LeLorier et al. NEJM 1997
20	Some Issues in the Comparisons of Meta-Analysis and Large Trial Ioannidis et al. JAMA 1998 Definition of large (arbitrary, power) Source of meta-analyses (why done?) Source of large trials Types of outcomes ( 1^o, 2^o) Meta-analysis statistics (FEM, REM) Definition of agreement (p-value, corr.) Reasons for disagreement
21
22	Meta-analyses vs. Mega-trials Cappelleri JC, Ioannidis JPA, deFerranti SD, Schmid CH, Aubert M, Chalmers TC, Lau J. Large trials versus meta-analyses of smaller trials: How do their results compare? JAMA 1996; 276:1332-38.
23	Large trials versus meta-analysis of smaller trials Data source
24	Large trials vs meta-analysis of smaller trials: How do their results compare ? By random effect calculations, agreements found between large and smaller trials in: 90% selected by sample size approach (1,000); 82% by statistical power approach Twice as many disagreements appeared when the variability among large studies and the variability among smaller studies was not considered (fixed effects calculations).
25
26
27	Large Trials vs Meta-Analysis of Smaller Trials: How do their results compare ? (cont.) Cappelleri et al, JAMA 1996 Of 15 disagreements between results of large and smaller trials using the random effects model, plausible explanations were identified in 10 meta-analyses: 5 with differences in the control rate between large and smaller trials 4 with specific protocol or study differences 1 with potential publication bias 2 other disagreements were not clinically important tentative reasons could be identified for 2 of the remaining 3 disagreements
28	Large trials vs meta-analysis of smaller trials: How do their results compare ? Meta-analyses of smaller studies are generally comparable with results from large studies. Differences can be attributed to insufficient sample sizes, control rates, or protocols. These reasons are not mutually exclusive. Publication bias is a possibility but has never been proven to be a factor. Need to explore reasons for heterogeneity.
29
30	Discrepancies between megatrials. Furukawa et al. J Clin Epidem 2000;53:1193-99. Why should large trials be the reference standard? What do we know about the agreements among large trials on the same problem?
31	Discrepancies between megatrials. Furukawa et al. J Clin Epidem 2000;53:1193-99. “megatrial” defined as >1,000 patients 289 pairs identified in Cochrane Library 79/289 (27%) pairs were statistically significantly different from each other 133 comparisons in LeLorier article 36/133 (27%)were statistically significantly different
32	Discrepancies between megatrials. Furukawa et al. J Clin Epidem 2000;53:1193-99. Agreement among megatrials was approximately as large as that reported between meta-analyses and megatrials If we were to base the recommendation for the treatment in question on the primary outcome, 53% (Cochrane set) and 31% (LeLorier set) of the treatment recommendation by a megatrial was not confirmed by a later megatrial. On the other hand, 30% to 47% of the treatments once found ineffective or harmful in a megatrial were shown to be beneficial by a later megatrial.
33	Insights from these empirical studies Heterogeneity of treatment effects is common among clinical trials, whether they are large or small; RCTs or observational studies Meta-analysis of small trials (dis)agree with large trials approximately as often dis(agreement) among large trials themselves We need to understand the cause of heterogeneity in clinical trials and learn how to handle them in meta-analysis
34	Controversy due to quality assessment: Screening mammography RCTs Gotszche and Olsen. Lancet 2000;355:129 A 1999 study found no decrease in breast cancer mortality in Sweden, where screening has been recommended since 1985 Reviewed methodological quality of mammography trials and repeated a meta-analysis
35	Controversy : Screening Mammography RCTs 8 trials identified Baseline imbalances were found in 6 of 8 trials 2 adequately randomized trials found no effect of screening on on breast cancer mortality pooled risk ratio 1.04 (95% CI 0.84 - 1.27) 6 inadequately randomized trials found significant effect Pooled risk ratio 0.75 (95% CI 0.67 – 0.83)
36	Relative risk of death from breast cancer in screening versus control groups
37	Mammography screening trials according to methodological quality
38	Definition of Poor Quality Based on Randomization adequacy Based on minor differences in mean age Failed to consider other explanations for difference in mean ages Failed to consider other measures of quality
39
40
41
42
43
44
45
46
47