1	New complexity: The ‘-omics’ revolution David F. Ransohoff, MD Departments of Medicine and Epidemiology University of North Carolina - Chapel Hill
2	New York Times, 2.3.04
3	Nature, 6.3.04
4	Science, 10.22.04
5	Organization 1. Lessons from history • evaluating diagnostic markers (‘rules of evidence’) 2. ‘Threats to validity’ from • chance • bias 3. Problems in current research • genomics - RNA arrays/BrCa prognosis • proteomics - serum/ovarian cancer screening 4. Lessons for future • exploring ‘-omics’ fields
6	History: Validation of cancer markers is ‘disappointing’ (not reproducible) 1. Non-invasive markers: Holy Grail of cancer diagnosis - CEA, CA19-9, CA125 - MRI of blood 2. CEA - initial report (PNAS): ~100% sensitivity, specificity for CRC - high expectations - disappointment when expensive ACS/CCS study did not reproduce initial results Disappointment would have been predicted and avoided if ‘rules of evidence’ were available.
7	History: Validation of cancer markers is ‘disappointing’ (not reproducible) 3. Lessons from CEA, for field of clinical epi/HSR - led to methodology to evaluate diagnostic tests -- rules of evidence -- concepts of bias, validation, ‘spectrum’ (Ransohoff and Feinstein. NEJM 1978) Methodology is still underdeveloped in 2005. 4 Past problems due to ‘culture clash’ - fields of laboratory medicine, clinical epidemiology have different ways of thinking, methods, rules of evidence, ‘Culture clash’ continues to be problem in 2005.
8	Present: Cancer markers are promising Knowledge of molecular biology provides targets to measure - past: knew little about what to target - now: know DNA ‘path’ from normal.. adenoma.. CRC Assays to measure targets - past: ‘one dimensional’ assays, like CEA, FOBT, PSA - now: multi-dimensional assays (measure almost any target) -DNA - primers and probes; amplify signal -protein - mass spectroscopy
9	Present: Cancer markers are promising But Mother Nature closely guards (her) secrets. New reductionist methods mean more data, but not necessarily more knowledge. Rules of evidence: not changed. Our job: -- to efficiently explore new technologies/fields -- to avoid predictable mistakes, inflated expectations. Exploration must be interdisciplinary, translational: molecular biology, clinical epidemiology, biostatistics
10	“Validation” 1. Definitions of “validation” are diverse, confusing. 2. But main concepts • are simple • can be usefully applied, right now
11	Nat Rev Cancer 2004;4:309-14
12	Three threats to validity 1. Chance Does chance explain ‘discrimination’? 2. Bias Does bias explain ‘discrimination’? 3. Generalizeability Does discrimination occur in clinically useful groups? (1) and (2) must be addressed in every study • or the study is not worth publishing • and (3) is not worth asking.
13	Chance as a threat to validity: one example (from many) - genomics
14
15
16
17	Editorialists and reviewers interpret results as “definitive” for clinical practice “... gene-expression patterns of primary tumours are better than available clinicopathological methods for determining the prognosis of individual patients.^6,10,11” Ramaswamy and Perou, Lancet 2003;361:1576-7 for biological research “... compelling evidence... genetic program of a cancer cell at diagnosis defines its biologic behavior many years later, refuting a competing hypothesis....” Wooster and Weber, NEJM 2003;348:2339-47
18	Chance (overfitting) can explain results Definition: In multivariable predictive models, overfitting occurs when a large number of predictor variables is fit to a small N of subjects. A model may ‘fit’ well or perfectly by chance, even if no real relationship. Simon, JNCI 2003 Consequence: results not reproducible in new set of data Method to check for: assess reproducibility of model in independent validation group (done <10% of reports. Ntzani, Lancet 2003)
19	Method to check for overfitting: assess reproducibility in independent sample Ransohoff. Nat Rev Cancer 2004
20	Did chance (partially) explain results of NEJM 2002? I.e. Are results not ‘definitive’? to the editor: “In research to validate a prognostic system, the inclusion of 61 patients [of the 295 in the ‘validation group’] from the original training group... [means] the validation group is not independent.... [and] the degree of prognostic discrimination may have been inflated....” NEJM 2003;348:1716.
21	Results not reproduced in independent group “Multi-center external validation study of the Amsterdam 70-gene prognostic signature: [are] results still out-performing the clinical-pathological criteria?” Piccart MJ, Loi S, Van’tVeer L, et al.; abstract at NSABP 12.8.04 Purpose: “to look at... utility... of the Amsterdam 70-gene prognostic signature (Van de Vijver et al, N Engl. J Med, 2002)... external, independent validation of the signature.” Results: “...overall performance of the 70-gene prognostic signature was inferior... compared to the original Amsterdam series” [emphasis added]
22	RNA Expression Arrays: Big Picture RNA expression and prognosis: Does it work? -It may work, in some tumors (e.g., lymphoma) -But perhaps not as well as claimed for tumors like BrCa that are heterogeneous (discuss Paik NEJM 2004; did avoid chance) 2. “Validation”: Is it necessary? -Yes; must be appropriately addressed in every study See, in press: Bias as a threat to validity of cancer molecular-marker research. Nat Rev Cancer 2005. Lessons from controversy: ovarian cancer screening and serum proteomics. JNCI 2005.
23	Chance as threat to validity: previous slides This is only one example of chance/ overfitting as ‘threat to validity’ in ‘-omics’ research. Are many other examples (Ntzani, Lancet, 2003); but problems will be easy to avoid, by checking for reproducibility in ‘independent’ group of subjects.
24	Bias as threat to validity: next slides In contrast, bias is: • less appreciated (and even less commonly addressed) • more difficult to handle (as in any observational epi.) Bias: systematic difference between compared groups, causing ‘erroneous’ comparison. (example...) Bias is a “plague upon the house of epidemiology.” (Cole) Biases are: • numerous (also no agreed-on nomenclature) • difficult to avoid • some are difficult to identify
25	Bias is so serious that results are guilty (of bias) until proven innocent. Proving innocence involves process: a. in design - to avoid bias b. in conduct - to measure if it occurred c. in interpretation - to determine if important Included in process also is to report a,b,c in detail in Methods, Results, and Discussion. See Ransohoff DF. Bias as a threat to validity of cancer molecular-marker research. Nat Rev Cancer 2005;5:in press
26	Bias as a threat to validity: one example (from many) - proteomics
27	Claims about serum proteomics to detect cancer are extraordinary • claims for multiple cancers (ovary, prostate, breast) -sensitivity: 95-100% -specificity: 95-100% • claims appear in Lancet, JNCI, WSJ, NBC, PBS, Redbook, etc. • Correlogic - patent filing for ‘pattern recognition’ process • LabCorp - license to use patented process in lab test
28
29	Proteomics Petricoin, Lancet 2.02 Purpose to diagnose ovarian cancer vs no cancer Methods • ovarian cancer, controls • serum assessed by mass spectroscopy (SELDI-TOF) • spectra analyzed by ‘genetic algorithm’ (Correlogic) Results ‘patterns’ discriminate
30	A TOF mass analyzer (Glish, Nat Rev 2003)
31
32	Proteomics Petricoin, Lancet 2.02 “The discriminatory pattern correctly identified all 50 ovarian cancer cases in the masked set... This result yielded a sensitivity of 100%… specificity of 95%…”
33	New York Times, 2.3.04
34
35	Does bias explain some serum proteomics results for ovarian cancer? (from news report of analysis of Keith Baggerly, Nature 04) Bias (one example) can occur if ‘signal’ is introduced by ‘run order’ of specimens on mass spectroscopy machine. E.g., If cancers and non-cancers are run on different days and different chips and if the mass spec machine ‘drifts’ over time, then non-biologic ‘signal,’ associated with Ca vs no-Ca, is hard-wired into results. Signal is not removed by ‘splitting’ sample into ‘training’ and ‘validation’.
36	Bias is the challenge in observational (non-experimental) research Bias is not ‘icing on the cake’; it is the cake. Bias is a large topic, difficult: -multiple biases; require different methods to address (e.g., randomization, blinding, uniform handling, etc) -some methods not available in observational research -some biases may be impossible to identify -even 1 bias may be fatal -bias, and process to address, is routinely ignored by authors, reviewers, editors in ‘omics’ research See Ransohoff, DF. Bias as a threat to validity of molecular-marker research. Nat Rev Cancer;2005;5:xx-yy
37	In RCT: how bias of possible ‘baseline inequality’ is assessed (usually as Table 1 in report of RCT, as in this example, NEJM)
38	In observational epidemiology ‘-omics’ study: how bias of possible ‘baseline inequality’ is assessed see Nat Rev Cancer 2005;5.
39	Organization 1. Lessons from history • evaluating diagnostic markers (‘rules of evidence’) 2. ‘Threats to validity’ from • chance • bias 3. Problems in current research • genomics - RNA arrays/BrCa prognosis • proteomics - serum/ovarian cancer screening 4. Lessons for future • exploring ‘-omics’ fields
40	Conclusion: Opportunities, challenges 1. An exciting era, because we: • know so much biology • have such powerful tools to measure biology 2. But rules of evidence have not changed. 3. The degree of hope and hype in 2005 is greater than results will likely support. 4. Disappointment may occur that, in retrospect, will have been predictable, and is due in part to culture clash. 5. We must improve scientific process (e.g., handle bias). 6. Our job: Go back to #1... and figure out how to avoid predictable disappointment and wasted effort... and how to generate useful knowledge about new markers.
41	References 1. Sullivan Pepe M, Etzioni R, Feng Z, Potter JD, Thompson ML, Thornquist M, et al. Phases of biomarker development for early detection of cancer. JNCI 2001; 93:1054-61. 2. Ransohoff DF. Developing molecular biomarkers for cancer. Science 2003; 299:1679-80. 3. Ransohoff DF. Rules of evidence for cancer molecular-marker discovery and validation. Nat Rev Cancer 2004; 4:309-14. 4. Ransohoff DF. Bias as a threat to validity of cancer molecular-marker research. Nat Rev Cancer 2005; in press. 5. Ransohoff DF. Lessons from controversy: ovarian cancer screening and serum proteomics. JNCI 2005; in press.