Intended for healthcare professionals

  1. Research
  2. Are experimental...
  3. Are experimental treatments for cancer in children superior to established treatments? Observational study of randomised controlled trials by the Children's Oncology Group
Papers

Are experimental treatments for cancer in children superior to established treatments? Observational study of randomised controlled trials by the Children's Oncology Group

BMJ 2005; 331 doi: https://doi.org/10.1136/bmj.38628.561123.7C (Published 01 December 2005) Cite this as: BMJ 2005;331:1295
  1. Ambuj Kumar, research associate1,
  2. Heloisa Soares, research associate1,
  3. Robert Wells, professor2,
  4. Mike Clarke, director3,
  5. Iztok Hozo, professor4,
  6. Archie Bleyer, professor2,
  7. Gregory Reaman, chairman5,
  8. Iain Chalmers, editor6,
  9. Benjamin Djulbegovic (djulbebm{at}moffitt.usf.edu), professor1
  1. 1 Department of Interdisciplinary Oncology, H Lee Moffitt Cancer Center and Research Institute, University of South Florida, Tampa, FL 33612, USA
  2. 2 University of Texas MD Anderson Cancer Center
  3. 3 UK Cochrane Centre, Oxford
  4. 4 Indiana University Northwest
  5. 5 Children's Oncology Group
  6. 6 James Lind Library, Oxford
  1. Correspondence to: B Djulbegovic
  • Accepted 16 September 2005

Abstract

Objectives To assess how often new treatments for childhood cancer assessed in phase III randomised trials are superior or inferior to standard treatments and whether the pattern of successes and failures in new treatments is consistent with uncertainty being the ethical basis for enrolling patients in such trials.

Design Observational study.

Setting Phase III randomised controlled trials carried out under the aegis of the Children's Oncology Group between 1955 and 1997, regardless of whether they were published.

Main outcome measures Overall survival, event free survival, and treatment related mortality.

Results 126 trials were included, involving 152 comparisons and 36 567 patients. The odds ratio for overall survival with experimental treatments was 0.96 (99% confidence interval 0.89 to 1.03), indicating that new treatments are as likely to be inferior as they are to be superior to standard treatments. This result was not affected by publication bias, methodological quality, treatment type, disease, or comparator.

Conclusions New treatments in childhood cancer tested in randomised controlled trials are, on average, as likely to be inferior as they are to be superior to standard treatments, confirming that the uncertainty principle has been operating.

Introduction

A letter to the BMJ suggested that potential participants of clinical trials have a right to know “all relevant details about the situation in which they find themselves.”1 In response, two paediatricians asked whether parents and children considering participating in Medical Research Council trials of new treatments for acute lymphoblastic leukaemia should be made aware that over the previous decade new experimental treatments have had better outcomes.2

In response, IC asked how likely it was that the successful run of new treatments was attributable to chance, and what should be someone's prior belief about the likelihood that the next new treatment proposed will also be an advance3; few data seemed to be available to tackle the latter question.3

We analysed a cohort of consecutive trials carried out by a common funder (the US National Cancer Institute) with the same platform for the development of preventive and therapeutic advances in oncology, to assess the probability that an experimental treatment for cancer in children will be superior to established treatments. We assessed how often new treatments for childhood cancer assessed in phase III randomised trials are superior or inferior to standard treatments.

We also asked whether the overall pattern of treatment successes could be predicted in advance, linking the ethical principle of equipoise or the “uncertainty principle” to the pattern.4–9 This principle states that the scientific and ethical justification for enrolment of patients into randomised controlled trials exists only if there is substantial uncertainty concerning which of the treatments is more likely to benefit patients.6–9 We previously hypothesised that there is a predictable relation between the uncertainty principle (the moral principle) on which trials are based and the ultimate outcomes of clinical trials.7 We formulated an “equipoise hypothesis,”7 postulating that a pattern of therapeutic advances is bounded by the equipoise or uncertainty principle—that is, if the uncertainty principle is observed, we would expect, over time, to find no significant difference between the proportion of trials that favour experimental treatments and those that favour standard treatments. We aimed to test this hypothesis.

Methods

We evaluated a consecutive series of all randomised phase III trials (126 trials involving 152 comparisons and 36 567 patients) started and completed between 1955 and 1997 under the aegis of the US National Cancer Institute sponsored Children's Oncology Group. We decided a priori to include all completed trials up to year 2000 (the last completed trial was started in 1997). These trials were published between 1960 and 2005. We obtained a list of trials from the National Cancer Institute; the list was verified by the Children's Oncology Group's headquarters, which also provided copies of the research protocols for each study. We analysed data from published and unpublished trials.

Collective uncertainty

We use the terms collective uncertainty and collective equipoise interchangeably.5 6 9 Approval of a Children's Oncology Group trial is a collective process. Before activation, the research protocols must pass a rigorous peer review process. They must be reviewed and approved by both the group and the National Cancer Institute. Although we could not elicit investigators' prior beliefs about the relative merits of the treatments to be compared, we can use the outcomes of the trials to assess whether they would have been right to have followed the uncertainty principle in carrying out these trials.10 11 Three possible relations exist between the trialists' uncertainties and the outcomes observed: on average new treatments are superior; on average standard treatments are superior; or on average there is no difference in outcomes between new and standard treatments. In light of our hypothesis, we postulated that the third possibility is most likely. This does not mean that we were seeking to show the existence of uncertainty in each trial. Rather, we looked for the existence of patterns over time in a large cluster of trials.11 12 We also examined the background to research protocols, which typically indicated that the researchers were not certain about the effect of the treatment in the trials.

Factors that may affect outcomes

Research during the past decade has identified that publication rate,13 methodological quality,14 and the choice of control interventions15 may affect a trial's results and their availability.

Publication rate

We used the National Cancer Institute definition of completed studies16 to determine the publication rate: a study is considered to have been completed if it has been closed to accrual, all patients completed therapy, and the study met its primary objectives. We excluded studies that were started but were closed early owing to poor patient accrual or had not yet completed follow-up and trials that compared two new treatments (n = 1). Trials that were closed early because the results clearly favoured one treatment over another were included in our analysis (n = 2). If a study had more than one publication we extracted data from the most recent report available at the time of our analysis.

Quality assessment

We extracted data on the methodological domains relevant to minimising bias and random error in the conduct and analysis of the trials.14 17 18 To ensure the accuracy of quality assessment of the trials, we used both the research protocols and the final publication for each trial.12 19 AK extracted the data, which were then reviewed by an independent investigator (HS). The principal investigator (BD) checked one in four trials at random and a paediatric oncologist (RW) checked one in 10 trials. Consensus meetings were held to resolve any interobserver differences. Interobserver agreements for quality appraisal and for assessment of treatment success were high (κ = 0.90 to 0.97).

Classification of comparator

The results of a trial may be affected by an inappropriate comparator,15 even if the study adheres to all contemporary standards of good design.17 In light of the evidence that trials using placebos or no therapy as controls may produce misleading results in favour of new treatments,11 20 21 we classified comparators in the trials as either active treatment or placebo or no active treatment, and we analysed these trials separately. In fact, violation of the uncertainty principle relates to the choice of an inferior comparator intervention.11 12 15 20 22

Distribution of outcomes between new and standard treatments

Using information provided in the background section to each paper and its related protocol, we judged what we considered was the new treatment and the standard treatment. The methods used to assess the superiority of experimental or standard treatments and the distribution of outcomes are reported in detail elsewhere.12 We assessed the proportion of trials that achieved a statistically significant difference according to the primary outcome specified a priori for each trial, and, using the categorisation suggested by Colditz et al,23 the preference between new and standard treatments as judged by the original investigators.

Furthermore, we used meta-analysis to derive overall estimates of the likelihood and size of any average differences between the effects of new and standard treatments. The unit of analysis was each randomised comparison within a trial. In trials with more than one comparison, we pooled all patients allocated to the new treatment when they did not have statistically superior outcomes to the patients in the standard group. If, however, one new intervention was classified as successful and others as not, then we compared this treatment only against the standard intervention, and we excluded from the analysis the remaining groups in the trial. Thus we biased our analysis against standard treatments, providing the best case scenario for concluding that new treatments are better.

We repeated the analysis for trials in which more than one new treatment was compared with standard treatment by using one of the new intervention groups only, selecting it at random, splitting the control group into the relevant number of subsets, and using each of these for comparison with each new treatment. Our results were not different in any important way, regardless of analytical method, and so we report only the initial, predefined analyses here.

In addition to pooling data on the primary outcomes as defined for each trial, we pooled data on overall survival and event free survival. These data were reported in most of the studies and represent the most important outcome in cancer. We also pooled data on treatment related mortality as described previously. Summary effects were expressed as hazard ratios or Peto odds ratios with 99% confidence intervals.24 25 We considered this the most appropriate way to minimise the chance of false positive results. Our prespecified meta-analytical technique for pooling the results of the trials was the assumption free method,24 but we also used a random effects model. The results remained virtually unchanged regardless of the methods used (see bmj.com).

We also evaluated the pattern of new successes over time. If a new success influences the outcome of another success then one would expect significant correlation between experimental treatments at time t and preceding times. If, however, the uncertainty principle is at work, then testing in each trial should be independent of previous testing. That is, a statistically significant autocorrelation would not be expected between treatment successes carried out at different times. Such a series of treatment successes would conform to a white noise pattern, with no significant autocorrelation in a time series analysis.

All analyses were done using STATA statistical software.

Results

See table on bmj.com for the characteristics of the 126 randomised trials in our analyses. Overall the methodological quality of trials was high.

Figure 1 shows the derivation of the 152 randomised comparisons that we analysed. The final analysis included 152 comparisons from 107 trials. Records were not available for 11 trials; these seemed to have been lost during frequent reorganisations of the parent organisations of the Children's Oncology Group (fig 1). These 11 trials totalled about 1750 patients—5% (1750/36 500) of the total number of participants analysed here—and therefore were unlikely to have an important effect on our findings. Primary outcomes were published for 99 (79%) of the 126 trials, and primary or secondary outcomes were published for 113 (90%) trials. We did not detect any differences in estimates of treatment effects derived from published and unpublished trials (see fig A on bmj.com).

Fig 1
Fig 1

Publication rate for phase III randomised clinical trials carried out by Children's Oncology Group

The documents we reviewed clearly indicated that, at the start of their studies, the Children's Oncology Group investigators were hoping that new treatments would be better than standard treatments. In their reports of completed studies, they stated preferences for new treatments in 47% of the comparisons and for standard treatments in 53% (P = 0.4). The investigators judged 26 (17%) interventions as breakthrough. (We arbitrarily defined as breakthrough interventions those interventions that were judged by the investigators highly preferred so they should become standard of care or that had an effect size so large that their log hazard ratio for survival or event free survival was - 1 or less.) Some new treatments were, however, worse than existing treatments. Only 29% of the trials achieved statistically significant differences in the investigators' a priori primary outcomes (see fig B on bmj.com).

Figure 2 shows the results of meta-analyses that take account of the size of each trial and time to event data. The Peto odds ratio for the principal outcome (overall survival) is 0.96 (99% confidence interval 0.89 to 1.04), indicating that new treatments were as likely to be inferior as they were to be superior to standard treatments (see fig C on bmj.com for forest plot showing results for individual trials). New treatments were slightly favoured for event free survival (0.91, 0.87 to 0.96), but this advantage was offset by increased treatment related mortality associated with new treatments (1.82, 1.21 to 2.75). This balance of benefits and harms is the likely reason why overall survival, on average, was similar for new and standard treatments (fig 2). We found no evidence of any autocorrelation over time between success in one trial and success in another (fig 3), suggesting that each trial represents an independent experiment in a given time with the aim of addressing the uncertainty that existed when the trial was designed.

Fig 2
Fig 2

Summary analyses according to predefined primary outcomes. Large squares indicate trials with more information, hence narrower 99% confidence intervals. When confidence interval crosses vertical line, result is not statistically significant (P<0.01)

Fig 3
Fig 3

Time series analysis of treatment effect (log hazard ratio) of studies carried out by Children's Oncology Group. White noise pattern indicates no significant autocorrelation between studies carried out at various time intervals. Log hazard <0 indicates superiority of new treatments and >0 a survival advantage for standard treatments

We did not detect any association between methodological quality or choice of comparator and the outcomes of trials. This is expected because of the high quality of these trials (see fig D on bmj.com). Neither did we detect any heterogeneity within our analyses of overall results by type of cancer studied. The largest single category related to trials for treatment of acute lymphoblastic leukaemia (odds ratio for survival 0.98, 0.92 to 1.08).

Discussion

New experimental treatments for childhood cancer assessed in phase III randomised trials are as likely to be inferior as they are to be superior to standard treatments. Our result also indicated unpredictability of individual trial results (fig 3). We believe that the pattern of successes and failures of new treatments is consistent with uncertainty being the basis for inviting patients to participate in such trials.

We further believe that this pattern of therapeutic success is a key reason behind major accomplishments in efforts to treat childhood cancer. The success has not come from a series of continuous, steady improvements, as selective reporting of treatment accomplishments may lead us to believe.2 On the contrary, our data show that outcomes of new treatments are as likely to be inferior as they are to be superior to standard treatments. The successful evolution of treatment resulted from empirical testing by investigators who acknowledged their uncertainty and chose to randomise between treatments, the relative effect of which they could not predict.7 12 This is the reason that the efforts of the Children's Oncology Group have been credited with dramatic achievements and improvements in outcomes of childhood cancers.26 27 This, of course, does not mean that improvements in the treatment of children with cancer have not arisen from efforts by others. For example, standard treatments for childhood acute lymphoblastic leukemia in 2005 is expected to result in greater than 80% cure rate, whereas the conventional treatments in the 1950s were associated with virtually a 0% cure rate.26

In our set of trials there were 26 breakthrough interventions (17%) as judged by the investigators. Some of these can be regarded as impressive: use of vincristine and actinomycin D in Wilms's tumour,28 the appropriate use of radiation therapy in Wilms's tumour,28 multiple versions of the leukaemia regimens applied to specific subgroups of patients with acute lymphocytic leukaemia,29 the use of autologous marrow transplant in neuroblastoma,30 the use of allogeneic bone marrow transplant in acute myeloblastic leukaemia in children,31 and the lack of utility of maintenance in acute myeloblastic leukaemia.32

More than 20 years ago Mosteller estimated that the public, sponsors of research, and investigators can expect that innovations will be successful about 50% of the time, which he called a “good investment.”33 In addition, over the past 25 years, studies of the track record of new treatments in other discipines have arrived at similar conclusions to ours.11 34–40 Similarly, studies investigating outcomes in cancer in adults, where progress has not been as dramatic as in childhood cancer, have also found a similar pattern of treatment successes.12 39 Thus, a pattern seems to be emerging: when the analyses are based on complete cohorts of published and unpublished trials, identified at inception, using meta-analysis to obtain estimates that take account of trial size and time to event data, half the time new treatments are either not different or superior to standard treatments or are not different or inferior to standard treatments.

All studies reported in the literature showing that innovative treatments are, on average, equally successful to standard treatments were, however, based on publicly sponsored trials. Industry sponsored trials are associated with increased likelihood of outcomes favouring sponsors, most likely due to selective reporting of favourable outcomes and violation of the uncertainty principle in the design of the trials.40

The pattern we describe holds only for the overall distribution of outcomes comparing new treatments with standard treatments. This fact is the key to preserving the clinical trial system, and the willingness of patients to participate in clinical trials, thereby contributing to further therapeutic advances. Provided the uncertainty principle applies, there is no a priori reason to be cautious about clinical trials in general, since new treatments tend to be, on average, neither better nor worse than standard therapies. If treatment success could consistently be predicted, patients would be expected to request those successful treatments, making enrolment into clinical trials and randomisation impossible. Particular treatments may prove to be better or worse than standard treatments but this will only be known after completion of the trial. Although data from model experimental systems and phase I and II trials of novel treatments may seem promising, they do not predict sufficiently well the outcomes of the phase III trials that will inform practice.41 In rare instances when treatment effects are dramatic (for example, imatinib in chronic myeloid leukaemia) careful control of biases using randomised trials may be unnecessary. But even under these circumstances, the uncertainty constraint would then shift to the design of phase I and II trials, and the question of the likelihood of innovative success would need to be assessed by examining how often drugs translated from animal research to humans lead to successful interventions without the need for phase III randomised trials.42

It is important to note that 94% of children diagnosed with cancer in the United States are seen at institutions that are members of the Children's Oncology Group.43 More than 60-65% of these patients are enrolled in the group's clinical trials.44 Therefore we believe that our results do seem to be generalisable to the pattern of treatment success seen in childhood cancer.

A possible limitation of our study is that we could not elicit researchers' prior beliefs about the relative merits of the treatments to be compared. However, the fact that we predicted a pattern of treatment success in advance before we collected data on treatment success45 provides strong corroborative evidence for our hypothesis. Ultimately, the question of which treatment is better needs to be answered by comparison. Such comparison needs to be based on the uncertainty principle whereby new treatments are always compared against the best available standard treatment. As new successful interventions are identified they in turn become the comparators in subsequent trials against which the new treatments that are continuously being introduced should be compared in randomised controlled trials.

What is already known on this topic

Potential participants in clinical trials should be given relevant details

In paediatric oncology this should include the track record of new treatments studied in earlier trials

No analyses exist of the average track record of new treatments for childhood cancer

What this study adds

Experimental treatments for cancer in children are as likely to be inferior as they are to be superior to standard treatments

The value of new experimental treatments cannot be confidently predicted in advance

Uncertainty has provided the ethical foundation for randomised trials and has been the driver for the substantial advances in several childhood cancers

We suggest that the results observed here and in related work12 39 are a consequence of a predictable relation between ethics and outcomes in randomised controlled trials. Our findings, and those from similar studies, should underpin the continuing need to resolve uncertainty through the randomised comparison of new and standard treatments. Over the past few decades the use of this principle of randomising when uncertain has served children with cancer well, it has been associated with striking improvements in the prognosis for some cancers. The use of the uncertainty principle provides the ethical foundation for randomised trials, and has been the driver for the substantial therapeutic advances in developing effective treatments for several childhood cancers. The scientific community and the public should be made more aware of how this mechanism underlies advances in clinical medicine (fig 4).

Fig 4
Fig 4

Proposed model of clinical discoveries showing how ethical principle converges to become scientific principle driving treatment progress. Graph insert shows actual distribution of treatment successes in trials carried out by Children's Oncology Group

Acknowledgments

We thank the Children's Oncology Group for their extensive cooperation, Craig Beam (Biostatistical Core, H Lee Moffitt Cancer Centre and Research Institute) for carrying out the time series analysis; William Dalton (H Lee Moffitt Cancer Centre and Research Institute), Steven Joffe (Dana-Farber Cancer Institute, Harvard Medical School), and Howard Mann (University of Utah, Salt Lake City) for their insightful comments and helpful feedback to an earlier version of this paper; Vineet Gupta (H Lee Moffitt Cancer Centre and Research Institute) for comments regarding the latest version of the manuscript; and the patients, health care providers, and others who took part in the clinical trials of the Children's Oncology Group and the legacy paediatric cooperative groups.

Footnotes

  • Embedded Image Description of studies and additional figures are on bmj.com

  • Contributors BD conceptualised the study, obtained funding as the direct response to IC's question about the probability of treatment success, and wrote the first draft of the paper along with AK. IC redrafted the paper, which was revised by all authors. AK extracted and analysed the data and helped with interpretation of data and writing the manuscript. HS and RW contributed to data extraction, data interpretation, and writing. MC helped with interpretation of data and writing the manuscript. IH contributed to statistical analysis and data interpretation. AB and GR helped secure access to data and, along with RW, provided unique content specific expertise in writing the paper. BD is guarantor.

  • Funding Research program on research integrity, Office of Research Integrity and National Institute of Health (grant No 1R01NS044417-01, NIH/ORI).

  • Competing interests Although some authors work within the Cochrane Collaboration and Children's Oncology Group, the views expressed here are their own and not necessarily those of the Cochrane Collaboration or Children's Oncology Group.

  • Ethical approval Not required.

References

References

  1. ↵
    1. Williamson C
    . Not gaining patients' consent in trials is deceitful. BMJ 1996; 312: 1479.
    OpenUrlFREE Full Text
  2. ↵
    1. Stephenson T,
    2. Walker DA
    . Gaining patients' consent. Telling parents all relevant details might reduce recruitment of children to trials. BMJ 1996; 313: 3623.
    OpenUrlFREE Full Text
  3. ↵
    1. Chalmers I
    . What is the prior probability of a proposed new treatment being superior to established treatments? BMJ 1997; 314: 745.
    OpenUrlFREE Full Text
  4. ↵
    1. Bradford Hill A
    . Medical ethics and controlled trials. BMJ 1963; 2: 10439.
  5. ↵
    1. Peto R,
    2. Baigent C
    . Trials: the next 50 years. BMJ 1998; 317: 11701.
    OpenUrlFREE Full Text
  6. ↵
    1. Freedman B
    . Equipoise and the ethics of clinical research. N Engl J Med 1987; 317: 1415.
    OpenUrlCrossRefPubMedWeb of Science
  7. ↵
    1. Djulbegovic B
    . Acknowledgment of uncertainty: a fundamental means to ensure scientific and ethical validity in clinical research. Curr Oncol Rep 2001; 3: 38995.
    OpenUrlCrossRefPubMed
    1. Edwards SJL,
    2. Lilford RJ,
    3. Braunholtz DA,
    4. Jackson JC,
    5. Hewison J,
    6. Thornton J
    . Ethical issues in the design and conduct of randomized controlled trials. Health Technol Assess 1998; 2: 1130.
  9. ↵
    1. Lilford RJ,
    2. Djulbegovic B
    . Equipoise and “the uncertainty principle” are not mutually exclusive. BMJ 2001; 322: 795.
  10. ↵
    1. Djulbegovic B,
    2. Bennet CL,
    3. Adams JR,
    4. Lyman GH
    . Industry-sponsored research. Lancet 2000; 356: 21934.
    OpenUrl
  11. ↵
    1. Djulbegovic B,
    2. Lacevic M,
    3. Cantor A,
    4. Fields KK,
    5. Bennett CL,
    6. Adams JR,
    7. et al
    . The uncertainty principle and industry-sponsored research. Lancet 2000; 356: 6358.
    OpenUrlCrossRefPubMedWeb of Science
  12. ↵
    1. Soares HP,
    2. Kumar A,
    3. Daniels S,
    4. Swann S,
    5. Cantor A,
    6. Hozo I,
    7. et al
    . Evaluation of new treatments in radiation oncology: are they better than standard treatments? JAMA 2005; 293: 9708.
    OpenUrlCrossRefPubMedWeb of Science
  13. ↵
    1. Dickersin K
    . How important is publication bias? A synthesis of available data. AIDS Edu Prev 1997; 9: 1521.
  14. ↵
    1. Egger M,
    2. Davey Smith G,
    3. Altman DG
    1. Juni P,
    2. Altman DG,
    3. Egger M
    . Assessing the quality of randomised controlled trials. In: Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in health care. Meta-analysis in context. London: BMJ Books, 2001: 87108.
  15. ↵
    1. Mann H,
    2. Djulbegovic B
    . Choosing a control intervention for a randomised clinical trial. BMC Med Res Methodol 2003; 3: 7.
    OpenUrlCrossRefPubMed
  16. ↵
    1. National Cancer Institute
    . Cancer trials. http://cancertrials.nci.nih.gov/clinicaltrials (acccessed 12 Nov 2005).
  17. ↵
    1. Altman DG,
    2. Schulz KF,
    3. Moher D,
    4. Egger M,
    5. Davidoff F,
    6. Elbourne D,
    7. et al
    . The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001; 134: 66394.
    OpenUrlCrossRefPubMedWeb of Science
  18. ↵
    1. Egger M,
    2. Davey Smith G,
    3. Altman D
    . Systematic reviews in health care. Meta-analysis in context. 2nd ed. London: BMJ Books, 2001.
  19. ↵
    1. Soares HP,
    2. Daniels S,
    3. Kumar A,
    4. Clarke M,
    5. Scott C,
    6. Swann S,
    7. et al
    . Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004; 328: 224.
    OpenUrlAbstract/FREE Full Text
  20. ↵
    1. Djulbegovic B,
    2. Cantor A,
    3. Clarke M
    . The importance of preservation of the ethical principle of equipoise in the design of clinical trials: relative impact of the methodological quality domains on the treatment effect in randomized controlled trials. Account Res 2003; 10: 30115.
    OpenUrlCrossRefPubMed
  21. ↵
    1. Rothman KJ,
    2. Michels KB
    . The continuing unethical use of placebo controls. N Engl J Med 1994; 331: 3948.
    OpenUrlCrossRefPubMedWeb of Science
  22. ↵
    1. Mann H,
    2. Djulbegovic B
    . Why comparisons must address genuine uncertainties: James Lind Library, 2005 (
  23. ↵
    1. Colditz GA,
    2. Miller JN,
    3. Mosteller F
    . How study design affects outcomes in comparisons of therapy. I: medical. Stat Med 1989; 8: 44154.
    OpenUrlPubMedWeb of Science
  24. ↵
    1. Early Breast Cancer Trialists Collaborative Group
    . Introduction and methods section. Treatment of early breast cancer. Worldwide evidence 1985–1990. Oxford: Oxford University Press, 1990.
  25. ↵
    1. Early Breast Cancer Trialists Collaborative Group
    . Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: an overview of the randomised trials. Lancet 2005; 365: 1687717.
    OpenUrlCrossRefPubMedWeb of Science
  26. ↵
    1. Simone JV
    . Childhood leukemia—successes and challenges for survivors. N Engl J Med 2003; 349: 6278.
    OpenUrlCrossRefPubMedWeb of Science
  27. ↵
    1. Wittes RE
    . Therapies for cancer in children—past successes, future challenges. N Engl J Med 2003; 348: 7479.
    OpenUrlCrossRefPubMedWeb of Science
  28. ↵
    1. D'Angio GJ,
    2. Evans AE,
    3. Breslow N,
    4. Beckwith B,
    5. Bishop H,
    6. Feigl P,
    7. et al
    . The treatment of Wilms' tumor: results of the national Wilms' tumor study. Cancer 1976; 38: 63346.
    OpenUrlCrossRefPubMedWeb of Science
  29. ↵
    1. Gaynon P,
    2. Steinherz P,
    3. Bleyer W,
    4. Ablin A,
    5. Albo V,
    6. Finklestein J,
    7. et al
    . Improved therapy for children with acute lymphoblastic leukemia and unfavorable presenting features: a follow-up report of the Childrens Cancer Group study CCG-106. J Clin Oncol 1993; 11: 223442.
    OpenUrlAbstract/FREE Full Text
  30. ↵
    1. Matthay KK,
    2. Villablanca JG,
    3. Seeger RC,
    4. Stram DO,
    5. Harris RE,
    6. Ramsay NK,
    7. et al
    . Treatment of high-risk neuroblastoma with intensive chemotherapy, radiotherapy, autologous bone marrow transplantation, and 13-cis-retinoic acid. Children's Cancer Group. N Engl J Med 1999; 341: 116573.
    OpenUrlCrossRefPubMedWeb of Science
  31. ↵
    1. Woods WG,
    2. Neudorf S,
    3. Gold S,
    4. Sanders J,
    5. Buckley JD,
    6. Barnard DR,
    7. et al
    . A comparison of allogeneic bone marrow transplantation, autologous bone marrow transplantation, and aggressive chemotherapy in children with acute myeloid leukemia in remission. Blood 2001; 97: 5662.
    OpenUrlAbstract/FREE Full Text
  32. ↵
    1. Wells RJ,
    2. Woods WG,
    3. Buckley JD,
    4. Odom LF,
    5. Benjamin D,
    6. Bernstein I,
    7. et al
    . Treatment of newly diagnosed children and adolescents with acute myeloid leukemia: a Childrens Cancer Group study. J Clin Oncol 1994; 12: 236777.
    OpenUrlAbstract/FREE Full Text
  33. ↵
    1. Mosteller F
    . Innovation and evaluation. Science 1981; 211: 8816.
    OpenUrlAbstract/FREE Full Text
  34. ↵
    1. Bubker JP,
    2. Barnes BA,
    3. Mosteller F
    1. Gilbert JP,
    2. McPeek B,
    3. Mosteller F
    . Progress in surgery and anesthesia: benefits and risks of innovative therapy. In: Bubker JP, Barnes BA, Mosteller F, eds. Costs, risks and benefits of surgery. New York: Oxford University Press, 1977: 12469.
    1. Colditz GA,
    2. Miller JN,
    3. Mosteller F
    . Measuring gain in the evaluation of medical technology. The probability of a better outcome. Int J Technol Asses Health Care 1988; 4: 63742.
    OpenUrl
    1. Gotzsche PC
    . Meta-analysis of NSAIDs: contribution of drugs, doses, trial designs, and meta-analytic techniques. Scand J Rheumatol 1993; 22: 25560.
    OpenUrlPubMedWeb of Science
    1. Chlebowski RT,
    2. Lillington LM
    . A decade of breast cancer clinical investigation: results as reported in the program/proceedings of the American Society of Clinical Oncology. J Clin Oncol 1994; 12: 178995.
    OpenUrlAbstract/FREE Full Text
    1. Machin D,
    2. Stenning S,
    3. Parmar M,
    4. Fayers P,
    5. Girling D,
    6. Stephens R,
    7. et al
    . Thirty years of medical research council randomized trials in solid tumours. Clin Oncology 1997; 9: 10014.
    OpenUrlCrossRef
  39. ↵
    1. Joffe S,
    2. Harrington DP,
    3. George SL,
    4. Emanuel EJ,
    5. Budzinski LA,
    6. Weeks JC
    . Satisfaction of the uncertainty principle in cancer clinical trials: retrospective cohort analysis. BMJ 2004; 328: 1463.
    OpenUrlAbstract/FREE Full Text
  40. ↵
    1. Lexchin J,
    2. Bero LA,
    3. Djulbegovic B,
    4. Clark O
    . Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003; 326: 116770.
    OpenUrlAbstract/FREE Full Text
  41. ↵
    1. Rothschild BB,
    2. King NM
    . Phase 1 clinical trials in oncology. N Engl J Med 2005; 352: 24513.
    OpenUrlCrossRefPubMedWeb of Science
  42. ↵
    1. Pound P,
    2. Ebrahim S,
    3. Sandercock P,
    4. Bracken MB,
    5. Roberts I
    . Where is the evidence that animal research benefits humans? BMJ 2004; 328: 5147.
    OpenUrlFREE Full Text
  43. ↵
    1. Ross JA,
    2. Severson RK,
    3. Pollock BH,
    4. Robison LL
    . Childhood cancer in the United States. A geographical analysis of cases from the Pediatric Cooperative Clinical Trials groups. Cancer 1996; 77: 2017.
    OpenUrlCrossRefPubMedWeb of Science
  44. ↵
    1. Bleyer WA,
    2. Tejeda H,
    3. Murphy SB,
    4. Robison LL,
    5. Ross JA,
    6. Pollock BH,
    7. et al
    . National cancer clinical trials: children have equal access; adolescents do not. J Adolesc Health 1997; 21: 36673.
    OpenUrlCrossRefPubMedWeb of Science
  45. ↵
    1. Lipton P
    . Testing hypotheses: prediction and prejudice. Science 2005; 307: 21921.
    OpenUrlAbstract/FREE Full Text
View Abstract
Back to top