Effects of preventive home visits to elderly people living in the community: systematic review

Dear Editor

As part of a teaching exercise for a session on an MPH course the
students and I recently examined the article by van Haastregt et al. This
was chosen at random, without any suspicion that it contained errors. To
illustrate problems associated with data extraction and analysis in
systematic reviews, we obtained the original papers for the 15 included
studies and looked in detail at two of the outcomes considered in the
original review - falls and mortality. These two outcomes were chosen for
reasons of feasibility. When we did this, considering the full data
presented in the original reports, we uncovered a pattern of small effects
that generally favoured the preventive home visit arms of the included
trials. This is illustrated for the mortality data in the odds ratio
diagram below, but also applied to data on falls:

http://www.bham.ac.uk/arif/preventive2.jpg

I would highlight three issues arising from the re-analysis
performed.

First, problems associated with vote-counting as a method of analysis
in systematic reviews were highlighted in correspondence following the
original article. This concern seems to be vindicated as the vote-counting
technique employed in the original review, seems to have over-looked the
possibility of drawing conclusions on the basis of consistent, small
effects, few of which are statistically significant. Quantitative summary
(meta-analysis) is the obvious way to identify such a pattern, but
qualitative synthesis which records and considers the direction of the
results and the sizes of any effects where quantified would also work.
There appears to be an urgent need to re-examine the results of all
outcomes of the included studies in the original systematic review using
such approaches.

Second, even ahead of such fuller re-analysis, the mortality data
alone seem to challenge the conclusion reached by van Haastregt et al, "No
clear evidence was found in favour of the effectiveness of preventive home
visits to elderly people living in the community". Although there is some
statistical heterogeneity in the mortality results, there is reasonably
clear evidence of a small beneficial effect on mortality in most trials
which demands that consideration continue to be given to preventive health
visits as a useful health intervention. Even if it is thought that the
effect of the given intervention on mortality is unlikely by virtue of
biological implausibility, the fact that the effect has been observed
empirically suggests further investigation is essential. This is
completely at odds with the bottom-line offered by van Haastregt et al
that, "It seems essential that the effectiveness of such visits is
improved, but if this cannot be achieved consideration should be given to
discontinuing these visits."

Finally, the case in question alerts to the fallibility of standard
practice in implementing the results of systematic reviews (and other
research). In this we have come to assume that provided articles meet
standard critical appraisal criteria they will provide internally valid
results which can be safely applied if benefits seem to outweigh
disbenefits and costs. Unfortunately the review in question meets commonly
used criteria for systematic reviews such as clarity of question, focus on
RCTs as the most appropriate study design to assess effectiveness,
comprehensiveness of search strategy and assessment of the quality of
included studies. The unwary might have been tempted to act on it. Thus,
particularly in the context of decisions with far-reaching consequences,
this example should remind that greater depth of assessment is essential.
In retrospect, the inability to follow the conclusions made in this
systematic review back to the results as they would have been presented in
the original papers should have raised an alarm. In our view the results
of the included studies as summarised were not a true reflection of the
data in the original papers. A simple safeguard that would help alert to
problems of this sort is to directly scrutinise a selection of the
included studies prior to decisions on implementation. We suggest that
particularly for health policy and population level health care decisions,
this step should become routine.

I have no conflicts of interest concerning this topic.