<< Back to Agenda

Slide 1: New complexity: The '-omics' revolution

David F. Ransohoff, MD
Departments of Medicine and Epidemiology
University of North Carolina - Chapel Hill

Slide 2: New York Times, 2.3.04

New Cancer Test Stirs Hope and Concern (Picture of newspaper article)

Slide 3: Nature, 6.3.04

Running before we can walk

Slide 4: Science, 10.22.04 (Picture of paper)

Getting the noise out of gene arrays (Picture of journal article)

Slide 5: Organization

1. Lessons from history

• evaluating diagnostic markers ('rules of evidence')

2. 'Threats to validity' from

• chance
• bias

3. 'Problems in current research

• genomics - RNA arrays/BrCa prognosis
• proteomics - serum/ovarian cancer screening

4. 'Lessons for future

• exploring '-omics' fields

Slide 6: History: Validation of cancer markers is 'disappointing' (not reproducible)

1. Non-invasive markers: Holy Grail of cancer diagnosis

• - CEA, CA19-9, CA125
• MRI of blood

2. CEA

• - initial report (PNAS): ~100% sensitivity, specificity for CRC
• high expectations
• disappointment when expensive ACS/CCS study did not reproduce initial results

Disappointment would have been predicted and avoided if 'rules of evidence' were available.

Slide 7: History: Validation of cancer markers is 'disappointing' (not reproducible)

3. Lessons from CEA, for field of clinical epi/HSR

• led to methodology to evaluate diagnostic tests
  • rules of evidence
  • concepts of bias, validation, 'spectrum'

(Ransohoff and Feinstein. NEJM 1978)

Methodology is still underdeveloped in 2005.

4. Past problems due to 'culture clash'

• fields of laboratory medicine, clinical epidemiology have different ways of thinking, methods, rules of evidence,

'Culture clash' continues to be problem in 2005.

Slide 8: Present: Cancer markers are promising

Knowledge of molecular biology provides targets to measure

• past: knew little about what to target
• now: know DNA 'path' from normal.. adenoma..CRC

Assays to measure targets

• past: 'one dimensional' assays, like CEA, FOBT, PSA
• now: multi-dimensional assays (measure almost any target)
  • DNA - primers and probes; amplify signal
  • protein - mass spectroscopy

Slide 9: Present: Cancer markers are promising

But Mother Nature closely guards (her) secrets.

New reductionist methods mean more data, but not necessarily more knowledge.

Rules of evidence: not changed.

Our job:

• to efficiently explore new technologies/fields
• to avoid predictable mistakes, inflated expectations.

Exploration must be interdisciplinary, translational: molecular biology, clinical epidemiology, biostatistics

Slide 10: "Validation"

1. Definitions of "validation" are diverse, confusing.
2. But main concepts

• are simple
• can be usefully applied, right now

Slide 11: Meanings of the Word "Validation (Picture) Nat Rev Cancer 2004;4:309-14

Slide 12: Three threats to validity

1. Chance Does chance explain 'discrimination'?
2. Bias Does bias explain 'discrimination'?
3. Generalizeability Does discrimination occur in clinically useful groups?

(1) and (2) must be addressed in every study

• or the study is not worth publishing
• and (3) is not worth asking.

Slide 13: Chance as a threat to validity: one example (from many) - genomics

Slide 14: NEJM (Picture of the Journal)

Slide 15: Gene Expression Signature as a Predictor of Survival in Breast Cancer (Picture)

Slide 16: B. All patients (graph)

Slide 17: Editorialists and reviewers interpret results as "definitive"

for clinical practice

"... gene-expression patterns of primary tumors are better than available clinicopathological methods for determining the prognosis of individual patients.6,10,11"

Ramaswamy and Perou, Lancet 2003;361:1576-7

for biological research

"... compelling evidence... genetic program of a cancer cell at diagnosis defines its biologic behavior many years later, refuting a competing hypothesis...."

Wooster and Weber, NEJM 2003;348:2339-47

Slide 18: Chance (overfitting) can explain results

Definition: In multivariable predictive models, overfitting occurs when a large number of predictor variables is fit to a small N of subjects. A model may 'fit' well or perfectly by chance, even if no real relationship. Simon, JNCI 2003

Consequence: results not reproducible in new set of data

Method to check for: assess reproducibility of model in independent validation group (done <10% of reports. Ntzani, Lancet 2003)

Slide 19: Method to check for overfitting: assess reproducibility in independent sample (Graph)

Ransohoff. Nat Rev Cancer 2004

Slide 20: Did chance (partially) explain results of NEJM 2002?

I.e. Are results not 'definitive'?

to the editor:

"In research to validate a prognostic system, the inclusion of 61 patients [of the 295 in the 'validation group'] from the original training group... [means] the validation group is not independent.... [and] the degree of prognostic discrimination may have been inflated...."

NEJM 2003;348:1716.

Slide 21: Results not reproduced in independent group

"Multi-center external validation study of the Amsterdam 70-gene prognostic signature: [are] results still out-performing the clinical-pathological criteria?"

Piccart MJ, Loi S, Van'tVeer L, et al.; abstract at NSABP 12.8.04

Purpose: "to look at... utility... of the Amsterdam 70-gene prognostic signature (Van de Vijver et al, N Engl. J Med, 2002)... external, independent validation of the signature."

Results: "...overall performance of the 70-gene prognostic signature was

• inferior... compared to the original Amsterdam series

" [emphasis added]

Slide 22: RNA Expression Arrays: Big Picture

RNA expression and prognosis: Does it work?

• It may work, in some tumors (e.g., lymphoma)
• But perhaps not as well as claimed for tumors like BrCa
• that are heterogeneous
• (discuss Paik NEJM 2004; did avoid chance)

"Validation": Is it necessary?

• Yes; must be appropriately addressed in every study

See, in press:

Bias as a threat to validity of cancer molecular-marker research. Nat Rev Cancer 2005.

Lessons from controversy: ovarian cancer screening and serum proteomics. JNCI 2005.

Slide 23: Chance as threat to validity: previous slides

This is only one example of chance/ overfitting as 'threat to validity' in '-omics' research.

Are many other examples (Ntzani, Lancet, 2003); but problems will be easy to avoid, by checking for reproducibility in 'independent' group of subjects.

Slide 24: Bias as threat to validity: next slides

In contrast, bias is:

• less appreciated (and even less commonly addressed)
• more difficult to handle (as in any observational epi.)

Bias: systematic difference between compared groups, causing 'erroneous' comparison. (example...)

Bias is a "plague upon the house of epidemiology." (Cole)

Biases are:

• numerous (also no agreed-on nomenclature)
• difficult to avoid
• some are difficult to identify

Slide 25: Bias is so serious that results are guilty (of bias) until proven innocent.

Proving innocence involves process:

1. in design - to avoid bias
2. in conduct - to measure if it occurred
3. in interpretation - to determine if important

Included in process also is to report a,b,c in detail in Methods, Results, and Discussion.

See Ransohoff DF. Bias as a threat to validity of cancer molecular-marker research. Nat Rev Cancer 2005;5:in press

Slide 26: Bias as a threat to validity: one example (from many) - proteomics

Slide 27: Claims about serum proteomics to detect cancer are extraordinary

• claims for multiple cancers (ovary, prostate, breast)
• -sensitivity: 95-100%
• -specificity: 95-100%
• claims appear in Lancet, JNCI, WSJ, NBC, PBS, Redbook, etc.
• Correlogic - patent filing for 'pattern recognition' process
• o
• LabCorp - license to use patented process in lab test

Slide 28: Mechanisms of Disease

Use of Proteomic patterns in serum to identify ovarian cancer (Picture reprint)

Slide 29: Proteomics Petricoin, Lancet 2.02

Purpose to diagnose ovarian cancer vs no cancer

Methods

• ovarian cancer, controls
• serum assessed by mass spectroscopy (SELDI-TOF)
• spectra analyzed by 'genetic algorithm' (Correlogic)

Results

'patterns' discriminate

Slide 30: A TOF mass analyzer (Glish, Nat Rev 2003) (Chart)

Slide 31: Chromatogram (Chart )

Slide 32: Proteomics Petricoin, Lancet 2.02

"The discriminatory pattern correctly identified all 50 ovarian cancer cases in the masked set... This result yielded a sensitivity of 100%… specificity of 95%…"

Slide 33: New York Times, 2.3.04 (Picture of page NY Times page reproduced)

Slide 34: Running Before we Walk? (Picture of page NY Times page reproduced)

Slide 35: Does bias explain some serum proteomics results for ovarian cancer?

(from news report of analysis of Keith Baggerly, Nature 04)

Bias (one example) can occur if 'signal' is introduced by 'run order' of specimens on mass spectroscopy machine.

E.g., If cancers and non-cancers are run on different days and different chips and if the mass spec machine 'drifts' over time, then non-biologic 'signal,' associated with Ca vs no-Ca, is hard-wired into results. Signal is not removed by 'splitting' sample into 'training' and 'validation'.

Slide 36: Bias is the challenge in observational (non-experimental) research

Bias is not 'icing on the cake'; it is the cake.

Bias is a large topic, difficult*

• -multiple biases; require different methods to address
• -(e.g., randomization, blinding, uniform handling, etc)
• -some methods not available in observational research
• -some biases may be impossible to identify
• -even 1 bias may be fatal
• -bias, and process to address, is routinely ignored by authors, reviewers, editors in 'omics' research

*See Ransohoff, DF. Bias as a threat to validity of molecular-marker research. Nat Rev Cancer;2005;5:xx-yy

Slide 37: In RCT: how bias of possible 'baseline inequality' is assessed (usually as Table 1 in report of RCT, as in this example, NEJM) (Chart)

Slide 38: In observational epidemiology '-omics' study: how bias of possible 'baseline inequality' is assessed see Nat Rev Cancer 2005;5. (Picture)

Slide 39: Organization

1. Lessons from history

• evaluating diagnostic markers ('rules of evidence')

2. 'Threats to validity' from

• chance

• bias

3. Problems in current research

• genomics - RNA arrays/BrCa prognosis
• proteomics - serum/ovarian cancer screening

4. Lessons for future

• exploring '-omics' fields

Slide 40: Conclusion: Opportunities, challenges

1. An exciting era, because we:
   1. know so much biology
   2. have such powerful tools to measure biology
2. But rules of evidence have not changed.
3. The degree of hope and hype in 2005 is greater than results will likely support.
4. Disappointment may occur that, in retrospect, will have been predictable, and is due in part to culture clash.
5. We must improve scientific process (e.g., handle bias).
6. Our job: Go back to #1... and figure out how to avoid predictable disappointment and wasted effort... and how to generate useful knowledge about new markers.

Slide 41: References

1. Sullivan Pepe M, Etzioni R, Feng Z, Potter JD, Thompson ML, Thornquist M, et al. Phases of biomarker development for early detection of cancer. JNCI 2001; 93:1054-61.
2. Ransohoff DF. Developing molecular biomarkers for cancer. Science 2003; 299:1679-80.
3. Ransohoff DF. Rules of evidence for cancer molecular-marker discovery and validation. Nat Rev Cancer 2004; 4:309-14.
4. Ransohoff DF. Bias as a threat to validity of cancer molecular-marker research. Nat Rev Cancer 2005; in press.
5. Ransohoff DF. Lessons from controversy: ovarian cancer screening and serum proteomics. JNCI 2005; in press.

<< Back to Agenda