More women are excluded from treatment arm of such trials because of cardiovascular events

Editor—Elina Hemminki and Klim McPherson studied the impact of postmenopausal hormone therapy on cardiovascular events and cancer.1 Meta-analysis has both potential and limitations,2 and the authors comment on bias introduced when data are pooled from clinical trials designed to measure something different. Data on cardiovascular events and cancer “were given incidentally,” mostly as “reasons for dropping out.” The authors conclude that pooled data do not support the notion that hormone replacement therapy prevents cardiovascular events, but a more honest interpretation is surely that a much higher proportion of women are excluded because of cardiovascular events from the treatment arm of such trials. Given the difficulty of blinding patients and the culture connecting hormones with blood clotting, this is not surprising.

The best information we have on the relation between hormone replacement therapy and mortality comes from the nurses' health study.3 In this study of over 120 000 women followed up since 1976, over 3600 women died, each of whom was matched with 10 controls. After adjustment for confounding variables, current hormone users had a lower risk of death; indeed, current hormone users with coronary risk factors had the largest reduction in mortality (relative risk 0.51; 95% confidence interval 0.45 to 0.57). While benefit decreased with long term use and was reversed more than five years after therapy (probably due to the increased risk of breast cancer and the “catching up” of postponed deaths), the overall impact of hormone replacement therapy on mortality was positive.

We calculated the numbers of deaths in two cohorts of 1000 women from age 55 (the average menopausal age), one of which took hormone replacement therapy for 10 years and was followed up for 10 more years, and the other of which did not take hormone replacement therapy. We used Northumberland's age specific death rates and relative risks from the nurses' health study

Although the relative risk of benefit was reduced during the second five years of therapy, the number of deaths prevented was more than in the first five years. Secondly, although in the final five years more deaths occurred in the cohort who had been given hormone replacement therapy, this group showed considerable benefit over the whole 20 year period. Thus the number needed to treat with hormone replacement therapy for 10 years to prevent one death was 54, compared with 70 for mild hypertension.4 Even if it did not benefit patients through good effects on menopausal symptoms and osteoporosis, hormone replacement therapy seems to be a good public health measure for women.

Search for studies was limited

Editor—Elina Hemminki and Klim McPherson's pooled analysis addresses the possible cardioprotective effect of hormone replacement therapy.1 We are concerned, however, that the headline results of the paper had the potential to mislead practitioners and patients. Our main concern was the use of results from short term trials of hormone replacement therapy to refute a hypothesis, the cardioprotective effect of such therapy, which was generated from long term observational epidemiological studies. This confusion between short and long term outcomes does not help in clinical decision making on the use of hormone replacement therapy.

Our specific concerns focus on the selection of studies for analysis, the ascertainment of end points, and the method of analysis. The search for relevant trials seems to have been restricted to Medline, specific languages, and a limited time period. Other authors suggest that Medline may contain only 30-80% of relevant trials on a particular subject.2 The reliance on published end points in the study meant that data on cardiovascular disease were not available for 13 of the 22 studies identified. Some of these data might have been available from direct contact with the authors. These factors are potential sources of bias in the pooled analysis which may be just as important as the possible selection bias in observational studies. Also, the validity of directly pooling studies that were heterogeneous in terms of intervention, population, and time for follow up was not addressed.

The main conclusions of the study were based on a P value of 0.04 from a test of significance on a hypothetical null hypothesis of an odds ratio of 0.7. The choice of null hypothesis is critical, and if an odds ratio of 0.8 had been chosen the results would not have been significant. This approach to analysis hides the fact that the sample size was too small to detect an odds ratio of 0.7. Our calculations suggest that a sample size of 50 000 would be required.

As the authors note, pooled analysis has the potential to enhance the usefulness of small trials. It is essential, however, that such analysis is conducted with adequate attention to the possible pitfalls and biases as these studies can greatly influence clinicians, decision makers, and the public.

Paper's main conclusion is no longer justified when data from all trials are considered

Editor—Elina Hemminki and Klim McPherson's meta-analysis concluded that hormone replacement therapy does not protect postmenopausal women against cardiovascular events.1 This finding is shocking: not only did it conflict with the results of large observational studies2 but the meta-analysis also managed to obtain such an apparently definitive conclusion from relatively few patients (4000) and cardiovascular events (17). These assertions seemed, however, to be based on odds ratios calculated without data from the most recent and largest trial.3 (Their table of odds ratios gave a total of 2859, while their abstract talked of 4124 patients. The difference is the exact number in the latest trial.) Repeating their analysis on the data from all the trials gave odds ratios of 0.95 (95% confidence interval 0.31 to 3.1) for cardiovascular events and 2.0 (0.23 to 45) for thromboembolic events. In contrast to the reported odds ratios of 1.39 and 2.89, these odds ratios were consistent with hormone replacement therapy reducing cardiovascular events by 30%. Hence the main conclusion of the paper is no longer justified when data from all the trials are considered.

Regardless of the actual odds ratios, however, I am concerned about the reliability of the data and the methodology used in the meta-analysis. Firstly, the study used data from trials designed to study outcomes other than cardiovascular disease and cancers. There is no guarantee that the reporting of adverse events in the hormone and control groups would be unbiased.

Secondly, only seven of the 23 trials reported any cardiovascular or thromboembolic events. Strictly, only these trials provided information on the effect of hormone replacement therapy in patients compared with a control group. However, data were analysed as if they came from one large trial, which they clearly did not.

Thirdly, the meta-analysis was prompted by the high incidence of cardiovascular events in one trial4; in fact, without this trial the meta-analysis contained too few events to provide any useful, additional, information on the issue.

Fourthly, the study included strokes among the cardiovascular events. Yet observational studies suggest little protective effect of hormone replacement therapy against stroke.2 Considering strokes and coronary heart diseases separately, however, again leaves too few events to be useful.

In conclusion, the meta-analysis provided no evidence to suggest that hormone replacement therapy does not protect postmenopausal women against cardiovascular events. Publication of this meta-analysis despite its low power, methodological weaknesses, and possible errors reinforces the need for caution and scepticism in interpreting results from meta-analyses of predominantly small trials,5 particularly those reporting controversial results.

Combining thromboembolic events with cardiovascular events does not support odds ratio of 0.7

Editor—One of the key messages in Elina Hemminki and Klim McPherson's paper was that there was insufficient evidence to suggest that postmenopausal hormone therapy prevents cardiovascular events.1 This conclusion, however, is based on the pooling of adverse thromboembolic events with cardiovascular events. The meta-analysis reported in table 2 of the paper shows that the odds ratio of a cardiovascular event alone was 1.39 (95% confidence interval 0.48 to 3.95), which does not rule out the true odds ratio of 0.7 (P=0.1); combining thromboembolic events with cardiovascular events raised the odds ratio to 1.64 (0.65 to 4.18), which does not support an odds ratio of 0.7 in the population (P=0.04). Thromboembolic events are a known side effect of hormone replacement therapy,2 and adding them to cardiovascular events would inflate the figures. In fact, the postmenopausal oestrogen/progestin interventions (PEPI) trial, which was quoted as an inspiration for the authors, shows only five events for cardiovascular disease compared with 10 for thromboembolic disease among the treatment groups.3 The nurses' health study of nearly 60 000 women studied over more than 600 000 person years has shown a large decrease in the risk of major coronary heart disease among women taking hormone replacement therapy.4 Hemminki and McPherson's paper fails to contradict that finding.

Inclusion of one particular study was inappropriate

Editor—We have concerns about the appropriateness of the methodology that Elina Hemminki and Klim McPherson used in their meta-analysis.1 The methods section describes 22 studies on 4124 women, but table 1 gives details of 23 studies on 4164 women. The main results (table 2) seem to relate to an unspecified number of studies on 2859 women. In that table the odds ratio for cardiovascular events is given as 1.39, which suggests that women who receive postmenopausal hormone therapy may be at increased risk and certainly do not have the benefit of a 0.3-0.7 relative risk, as many authors have claimed.2 3 4

The three cardiovascular events attributed to the 1973 paper by Aitken et al5 relate to a high dose of the oestrogen mestranol, which has not been used for over 20 years. Mestranol is generally agreed to be unacceptable for use postmenopausally. In Aitken et al's study, women underwent oophorectomy premenopausally and were recruited after a variable period ranging from six weeks to six years; although they were matched to the placebo group for height and weight, they were not matched for cigarette smoking or concomitant drug treatment. Smoking is fundamental here since it is particularly harmful to users of mestranol even at young ages.

Non-selective analysis of all the studies in table 1 gives an odds ratio of just under 1.0, and if Aitken et al's study is removed the odds ratio approaches 0.7. Clearly, little can be said with certainty if these studies are examined in isolation.

In making these calculations of different odds ratios we have followed the methodology adopted in the paper of summing the events and numbers of women across the studies. This, however, is a non-standard method that has the potential to be misleading. The more usual approach is to maintain the distinctness of the studies and to pool their findings by using a Mantel-Haenszel estimate or similar technique. This approach is preferable since it assumes only that the odds ratios are constant across studies; the absolute risks of a cardiovascular event may differ. The authors' method of analysis makes sense only if the studies are similar in all respects, such as inclusion criteria, doses, and length of follow up.

The authors seem to have ignored the considerable volume of recent literature, which includes not only epidemiological data but also other direct intervention studies.

The selected studies contain little information about the real impact of postmenopausal hormone therapy on cardiovascular events. Given the impact of such evidence based conclusions, it is disappointing that the methodology was not more rigorous.

Study's conclusions were incorrect

Editor—After pooling data from randomised clinical trials examining the effect of postmenopausal hormone replacement therapy on cardiovascular events and cancer, Elina Hemminki and Klim McPherson concluded that the “data do not support the notion that postmenopausal hormone therapy prevents cardiovascular events.”1 This conclusion is at odds with multiple previously published meta-analyses of observational studies. After reviewing the original studies that were pooled in this analysis we have several concerns regarding both the methods applied and the interpretation of the results. The authors did not use the standard accepted techniques for performing meta-analyses and for pooling studies, and they did not consider differences in dosing regimens and duration of follow up. In calculating the odds ratio for cardiovascular events related to hormone use they included numerous “non-informative” studies that contained no information about whether patients, in fact, experienced such an event. Additionally, two of the 12 reported cardiovascular events among users of hormone replacement therapy were misclassified: one was an episode of palpitations associated with chest pain without myocardial infarction, and the other was an episode of superficial thrombophlebitis.2 Finally, the authors compared their calculated odds ratio for cardiovascular events (which includes cerebrovascular events) with the odds ratio for coronary heart disease (which does not) from previous studies, blurring the distinction between cardiovascular events and coronary heart disease.

We repeated their analysis using their pooling methodology, but we corrected the two misclassifications and limited the at risk group to those studies that specifically noted the presence or absence of cardiovascular events (decreasing the hormone replacement therapy arm from 1818 to 1142 and the placebo arm from 1041 to 497). For comparison, a recent meta-analysis of observational studies found a relative risk of 0.65 of hormone replacement therapy on coronary heart disease,3 identical with that found in the current dataset (1).

Even though the odds ratios in this analysis are based on a small number of coronary heart disease events (six among 1142 users of hormone replacement therapy and four among 497 non-users) and the confidence intervals are broad, there is a 71% probability that the true odds ratio for hormone replacement therapy on coronary heart disease is <1.0 and a 53% chance that it is <0.7. Our concern is that the authors' conclusions may be given undue weight by doctors and women, thus denying some women the potential benefit of hormone replacement therapy. Just as selection bias may affect results from observational studies, the lesson to be learnt here is that misreading and misclassifying data from randomised trials can lead to incorrect conclusions.

Author's reply

Editor—The purpose of our work was not to make a synthesis of all the evidence concerning benefits and harms of postmenopausal hormone therapy. Its focus was narrower: simply to examine what can be learnt from published randomised trials of such therapy. The nurses' health study potentially suffers from possible selection bias, as do all non-experimental studies.Our study has its own problems, as discussed in our paper and in the letters by Sunil Shah and Leonaura Rhodes and by Valerie Seagroatt. However, the literature search was more extensive than Medline alone, as suggested by Shah and Rhodes. We did consider contacting the authors of papers not reporting adverse effects. Such an approach also has its problems, including selective response and costs, but it would clearly be worth while. We did not test whether the calculated odds ratios were significantly different from unity (as suggested by Shah and Rhodes); instead we tested how likely it is to find such odds ratios by chance, if the true ratio was 0.7 or 0.5—more sensible null hypotheses.

We grouped different types of serious (potentially life threatening) cardiovascular diseases, because for decisions whether to use postmenopausal hormones this is sensible. We kept superficial phlebitis and thrombophlebitis separate, because they have different clinical implications and are likely to be diagnosed and reported less consistently. The PEPI trial had only four thromboembolic events, not 10 as cited by Torbjorn Sundkvist (six were phlebitis).

The allocation method in Aitken et al's study is not clear, but random allocation is more likely than matched. As Farook Al-Azzawi and colleagues say, it concerned women who had undergone oophorectomy and took an atypical hormone, but it is included in our paper. We did not misclassify two cases, as suggested by Nananda F Col and colleagues. The three cases from the study by Aitken et al included in table 1 comprised one death from cerebrovascular catastrophe (reported in methods), one case of transient hemiparesis, and one case of prolonged chest pain requiring treatment (termed ischaemic attack in a footnote to table 1). Exclusion of this study did not change our results.

Seagroatt and Al-Azzawi and colleagues wonder what the denominators were. The study by Speroff et al contributed only to the odds ratios of breast cancer, because it was not specific for other outcomes (see methods). It certainly would have been clearer to have a footnote to this in table 2. One cannot calculate odds ratios for cardiovascular diseases as suggested by Seagroatt or Al-Azzawi and colleagues. Col and colleagues seem to have interpreted non-informative studies differently from us, but they do not specify the trials they have excluded. Since we excluded none that fitted our criteria we still are more confident with our analysis than theirs. Reducing the denominators, unnecessarily excluding the two cardiovascular events, and breaking the outcomes into small subgroups seem to explain their different results.

We found two new trials only after we had completed the final manuscript. Unfortunately, when we added their data we failed to correct a few numbers. The correct number of trials is 23, and the number of women is 2899 without Speroff et al's study and 4164 with it. These inaccuracies do not influence our results, but we apologise for the confusion they caused for readers.