Influence of trial sample size on treatment effect estimates: meta-epidemiological study
BMJ 2013; 346 doi: https://doi.org/10.1136/bmj.f2304 (Published 24 April 2013) Cite this as: BMJ 2013;346:f2304
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
To the Editor:
Dr. Dechartres and colleagues demonstrated that treatment effect estimates differed within meta-analyses solely based on trial sample size, with stronger effect estimates seen in small to moderately sized trials than in the largest trials.(1) In fact, we have criticized the controversy which was similarly found in the volume-based, quality analysis. Previously, Joynt et al analyzed Medicare fee-for-service patients with a primary discharge diagnosis of congestive heart failure (CHF) in the United States and found that in the low-volume group, being admitted to a hospital with a higher case volume was associated with lower mortality, lower readmission, and higher costs.(2) However, we questioned that the small sample size in the denominator in all low-volume hospitals may augment the effector.3 For example, there were in average 85, 284 and 602 patients discharged from low-volume, medium-volume and high-volume hospitals respectively in a month. If those receiving intensive cardiac care were the population at highest risk of death, there would be about 14 (16%), 105 (37%), and 415 (69%) persons at risk in low-volume, medium-volume and high-volume hospitals respectively. By chance, 1-2 patients (7.1-14.3%) may die in low-volume hospitals (11-13% reported for low-volume hospitals in the paper), 11-12 patients (9.5-11.4%) may die in medium-volume hospitals (10.5-11%) and 41-45 patients (10.5-10.8%) may die in the large-volume hospitals (10-10.5% reported for medium-and large- volume hospitals in the paper).(2, 3)
After risk adjustments, the death rate would be higher in low-volume groups because the change of numerator dramatically affected the results and closely approached to the mean of death rate on standard treatment for patients with CHF. According to the experience, we believed that the influence of sample size may differ with volume-based trials in the meta-analysis as well.
Reference
1. Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ. 2013; 346:f2304.
2. Joynt KE, Orav EJ, Jha AK. The association between hospital volume and processes, outcomes, and costs of care for congestive heart failure. Ann Intern Med. 2011; 154:94-102.
3. Lin GM, Li YH, Hospital volume and processes, outcomes, and costs of care for congestive heart failure. Ann Intern Med. 2011; 155:202
Competing interests: No competing interests
Re: Influence of trial sample size on treatment effect estimates: meta-epidemiological study
To the editor:
There is a third reason why smaller sample size might lead to the larger estimates of treatment effect described by Dechartres et al (1).
In our field (of respiratory medicine), meta-analyses have to synthesize data from heterogenous trials with significant variety in dose, drug within a class, duration of treatment and follow-up, and even outcomes.
Sample-size calculations are such that sample size is smaller when larger treatment effects are anticipated and vice versa. This is at odds with any implication that statistical significance is all and suggests smaller studies may be superior in focussing at the design stage on greater clinical significance.
We would agree with the authors that meta-analyses deserve more subtle interpretation, but suggest this should be to identify factors in smaller samples that account for stronger effects, rather than assuming larger trials are intrinsically better.
Reference
1. Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ. 2013; 346:f2304.
Competing interests: No competing interests