Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial

We have enjoyed reading the debate surrounding the REEACT trial, especially since it raised some interesting unresolved issues in cCBT research (e.g. the predominance of developer-led trials, and the unclear effectiveness of the popular and free-to-use MoodGYM programme). Of high relevance to this debate, we conduced a meta-analysis of MoodGYM, which is has just been published in the Australian and New Zealand Journal of Psychiatry: http://anp.sagepub.com/content/early/2016/07/05/0004867416656258.abstract It is worth noting that we were not in any way involved in the development of MoodGYM and we do not benefit from its success.

In brief, our main findings were as follows:

Comparisons from 11 studies demonstrated MoodGYM’s effectiveness for depression symptoms at post-intervention, with a small effect size (g = 0.36, 95% confidence interval: 0.17–0.56; I2 = 78%). Removing the lowest quality studies (k = 3) had minimal impact; however, adjusting for publication bias reduced the effect size to a non-significant level (g = 0.17, 95% confidence interval: −0.01 to 0.38). Comparisons from six studies demonstrated MoodGYM’s effectiveness for anxiety symptoms at post-intervention, with a medium effect size (g = 0.57, 95% confidence interval: 0.20–0.94; I2 = 85%). Both the type of setting (clinical vs non-clinical) and MoodGYM-developer authorship in randomised controlled trials had no meaningful influence on results; however, the results were confounded by the type of control deployed, level of clinician guidance, international region of trial and adherence to MoodGYM.

We hope that our paper is of interest to cCBT research stakeholders and that our findings have added value to the debate surrounding the issues raised by the REEACT trial. We would like to highlight that MoodGYM-developer authorship did not have a meaningful impact on results, and this shows that assumed ‘conflicts of interests’ are not always indicative of bias.

Many thanks

Conal Twomey & Dr Gary O'Reilly

We thank Mr Drysdale and colleagues for their response.

At the risk of an extended correspondence on these issues which may imply we are questioning the value of their work, we would make the following final points.

We accept that our reporting of the change in the primary depression outcome in the BMJ paper could have been better. Whether our error is fairly labelled with the stark term ‘outcome switching’ is open to debate, but we accept that, by rule, we have failed to be entirely transparent and we meet their criteria for such a rating. The BMJ has added additional information for readers (http://www.bmj.com/content/352/bmj.i195) and we have learned a valuable lesson for the future – which we agree is a positive outcome of their project. There was complete transparency in relation to protocol changes in our full report (freely available in the public domain) where we devoted a whole chapter to the subject.

We do not accept their response in terms of the EQ5D.

They state ‘publishing quality of life data in a separate academic paper is in no sense a recognised universal convention’.

We did not suggest it was a universal convention – just that it was ‘not unusual’.

They state that ‘we therefore think that readers should have been told that an extra three secondary outcomes were pre-specified and that there was a plan to report these elsewhere’.

The BMJ report does clearly state that ‘we present the results of a full economic evaluation elsewhere’ (reference 32). That could have been more specific (again, a useful learning point), but it is explicit that the BMJ report is not a complete report of outcomes. A casual reader might be forgiven for not wanting to follow that up in interests of time. To ignore that clear reference to other data when judging reporting of outcomes seems odd.

They state that ‘Expecting all readers to know that they should also be reviewing an additional set of documents - which may or may not exist, for a subset of trials depending on funder, with an unspecified time delay - represents an even more impractical method to ensure transparent outcome reporting’

Mr Drysdale and colleagues paint a compelling picture here. However, that picture is not very convincing in the case of REEACT. The existence of the economic outcomes in another document is clearly communicated (a journal title and a designation of ‘in press’ is the maximum reference detail that we were able to provide at that time). It in no way suggests the kind of complex detective work they imply. A simple email to the corresponding author could have cleared that up very quickly – readers will note that our team responded very quickly to their initial letter.

We note that the COMPARE website refers to outcomes reported in a 'final paper', while their BMJ response refers to a 'primary academic publication'. Does the BMJ paper on clinical outcomes meet the criteria for either? Both terms are ambiguous. If they had asked, we would have recommended they treat the final HTA report (published a matter of weeks after the BMJ) as the definitive source, as it is unrestricted in length and includes the maximum level of detail. Again, an email would have been sufficient to clear this up – hardly ‘impractical’, and a reasonable expectation given the serious concerns they are raising.

Again we note that the HTA report contains a whole chapter detailing the inevitable protocol changes that were made in the course of a five year project working under the NHS research governance framework. Maximum transparency; public domain; open access; no switching of outcomes. We also provided everything that was asked of us by the editors of the BMJ in the pre-publication process. Thus our disquiet with their ‘rush to judgement’ on the basis of the BMJ paper alone remains.

We have great sympathy that the COMPARE process may be onerous. As researchers, we are very aware of the pressures of workload and the need for pragmatic decisions to make projects feasible.

However, REEACT l is still assessed as ‘60% of pre-specified outcomes reported’ (as of 14/1/2016) on a public website ‘tracking switched outcomes in clinical trials’. We think that is a serious charge which is neither entirely fair nor accurate given the information in the BMJ paper and the clarifications about the EQ5D which we have provided. Having endured the more onerous task of delivering the REEACT trial (we calculated over 100,000 person-hours), they can perhaps understand that our sympathy with the time requirements of their system is more limited

We wish the team luck with the remainder of their project.

We thank Henry Drysdale and colleagues for their interest in our trial. They correspond on behalf of the COMPare project team who are ‘analysing each trial for outcome switching, by comparing the clinical trials registry (or ideally a trial protocol)’ [1].

There is no inconsistency between our trial report and our pre-specified trial protocol which has been in the public domain for several years [2]. The transparency of our reporting exceeds all CONSORT requirements. We recorded depression severity using the PHQ9 which we analysed as both a dichotomous outcome (as validated against ICD criteria) and as a continuous outcome. The primary trial endpoint was a dichotomous outcome derived from the PHQ9 at four months, and we also reported this outcome at 12 and 24 months. We specify our primary outcome as the PHQ9 on the trial registry, where there is also a link to our full trial protocol (exceeding the level of reporting required by the ISRCTN trial registry, and representing the ‘ideal’ of the COMPare project). For the avoidance of doubt we have reproduced the relevant excerpt from the trial protocol which is presented within the ISRCTN registry [2] and included in appendix 2 of the BMJ report [3]. ‘Primary outcome measure: our primary outcome will be depression severity and symptomatology as measured by a validated self-report measure (the Patient Health Questionnaire-9) at four months. The PHQ9 is a nine-item questionnaire, which records the core symptoms of depression. There are extensive US and non-US validation and sensitivity to change data. It has most recently been validated in a UK primary care population…….The primary outcome, depressed/not depressed (at four months) will be used in a logistic regression model to compare each of the computerised packages with usual care alone.’ In the trial registry and trial protocol we also specified the PHQ9 at 12 and 24 months as secondary outcomes.

Henry Drysdale and colleagues highlight that we omitted to present one of our secondary outcomes (EQD – the ‘EuroQoL’; our chosen measure of quality adjusted life years or ‘QALYs’). The BMJ report presented the full clinical outcomes of the REEACT trial [3] and we were not asked by the editors or peer reviewers to present the results of the full economic evaluation (where QALYs and incremental cost effectiveness ratios are the metric of interest). This is not an omission on our part, but reflects a separation of clinical and economic results which sometimes happens (even in the BMJ). Far from seeking to omit this secondary outcome, we currently have a paper in preparation to report the economic dimension of cCBT. Should the BMJ wish to publish this, we will be happy to oblige.

The only deviation we can see from the ISCRTN entry is the fact that we exceeded our initial trial sample size (691 in the published report versus 600 in the trial registry). We don’t think this is a hanging offence, and we did this to ensure we maintained our level of pre-specified statistical power when follow up was a little lower than we anticipated (such things do happen). We note that trials commonly fail to achieve their pre-specified sample size and Mr Henry Drysdale and colleagues [writing as members of the Centre for Evidence Based Medicine] failed to find anything positive to say about our trial in this [or any other] respect.

We do not know whether the correspondents have extensive experience of HTA funding and publication policies, but all their trials require a full NIHR report where all outcomes will be presented in their entirety, after publication of other papers. Splitting clinical and economic outcomes is also not unusual. Although we applaud their sterling work, they might wish to consider that trial publications should really be considered in their entirety in the COMPare process. The vagaries of publications schedules means that they may need to give authors a little more time before potentially rushing to judgement about 'missing outcomes' that are 'in press' in the publication machine. A full report commissioned by the NIHR Health Technology Assessment programme is now published in the NIHR Library and we commend this report to interested readers. We include the results of the clinical, economic and quantitative evaluations of cCBT derived from the REEACT trial [4]. We wish Mr Drysdale and colleagues well with the COMPare project and thank them for the opportunity to respond to their query.

[1] COMPare project website www.COMPare-Trials.org

[2] Full REEACT trial protocol at http://www.nets.nihr.ac.uk/__data/assets/pdf_file/0003/51438/PRO-06-43-0...

[3] Gilbody S et al, Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial), BMJ 2015;351:h5627.

[4] Littlewood E, Duarte A, Hewitt C, Knowles S, Palmer S, Walker S, et al. A randomised controlled trial of computerised cognitive behaviour therapy for the treatment of depression in primary care: The Randomised Evaluation of the Effectiveness and Acceptability of Computerised Therapy (REEACT) trial. Health Technology Assessment 2015; Volume 19 (Number 101).

Response by Gilbody and colleagues on behalf of the REEACT collaborative

We thank all the correspondents who took the time to read and comment on the REEACT trial.1 Our trial has provoked important discussions around the place of computerised therapy in modern care systems and the challenges of researching such interventions. Our trial is not the last word on cCBT but does illustrate that the value of computer delivered psychological therapy cannot be assumed and that evidence of efficacy does not necessarily translate into real world effectiveness. The REEACT trial demonstrated that two computer programmes that are still routinely offered in UK NHS primary care mental health services and which are recommended by NICE did not, on average, show any benefit for people with depression when added to routine NHS primary care.

Several correspondents recognised the value of the REEACT trial but some important issues were raised. Space prevents us from responding to each, but there were a number of points which deserve some comment and response.

Efficacy versus effectiveness
Several correspondents would have preferred if we had conducted another efficacy study, rather than a test of real world effectiveness.2 There is ample randomised evidence to show that cCBT can be efficacious when delivered under ideal conditions; when compared to nothing at all or when delivered with very high levels of support (often guided by experienced psychological therapists).3 Our trial adds little to this body of literature, but there remain important uncertainties relating the real world effectiveness of cCBT and to which REEACT speaks. The majority of patients with depression are treated entirely in primary care, and the REEACT trial is one of the first (and certainly one of the largest) trials to test the effectiveness (rather than efficacy) of this technology in a primary care setting.

For GPs, patients and commissioners of services a very important question is ‘what is the benefit I can expect to see if a person with depression is offered cCBT in addition to the usual care which they already receive?’ This is a question relating to effectiveness rather than efficacy and was the question posed to the UK research community by the body which commissioned the REEACT trial – the National Institute of Health Research (NIHR) Health Technology Assessment Programme. We think our results are informative and a number of criticisms relate to the choice of a pragmatic rather than explanatory design and the primary care setting of our trial. We defend our research in the face of this criticism.

Lack of engagement with cCBT
Several correspondents have sought to explain away the negative results of the REEACT trial with reference to a number of factors which were inherent in our pragmatic design. People consented to partake in a trial of cCBT and we can therefore assume that they were interested in cCBT as a therapeutic option. We sought to engage people with weekly phone contact and reminders to use the programmes. The vast majority logged on to the systems to look at the programmes. We know this since we were able to check computer records. However very few people returned to complete a second session, and the proportion of people who completed all sessions was very small indeed. We maintain this is a very important finding and the most significant contribution of the REEACT trial. We note that none of the correspondents suggested that the level of support that was offered in the REEACT trial was atypical of that which is offered in routine NHS practice and to that end we succeeded in delivering a fair test of real world effectiveness. Our research suggests that a higher level of support than was delivered in REEACT is needed to enhance uptake. This would require a re-design of current NHS services and a level of support that is greater than that which is recommended in the two programmes that were trialed. The effectiveness of enhanced levels of support needs to be tested in large scale effectiveness trials and we have our own programme of ongoing research which speaks to this issue. In the meantime we can be confident that cCBT as delivered in UK primary care did not demonstrate uptake or effectiveness. This remains an important finding.

Cross over and dilution of effect
One explanation for the negative result offered by some correspondents was the reported use of computer resources by a small proportion of participants in the usual care arm, which might have diluted the effect of cCBT. We think this is a red herring and a diversion from the finding of lack of engagement. There would needed to have been a very substantial cross over with high levels of engagement in the usual care arm, and a very powerful effect from only a very small number of sessions for cCBT to have been effective, but for this effect to have been diluted by crossover. We do not think this explanation stands up to scrutiny, and crossover effects are to be expected in pragmatic trials. The effects of cCBT were likely to be minimal in both intervention and usual care arms based on the lack of engagement alone. Invoking Occam’s razor, this is a more parsimonious explanation.

Compared to what?
REEACT was an effectiveness trial and is one of the first trials to evaluate cCBT in terms of the actual value that can be expected under real world conditions and when compared to usual GP care. Several correspondents commented that the outcomes under usual care make it difficult to demonstrate the benefits of cCBT, and would have preferred a ‘do nothing’ comparator arm. Such studies have already been undertaken and we have noted the results of these, but they are not a sufficient level of evidence in establishing the value of new technology. Demonstrating effectiveness is always more difficult than demonstrating efficacy, but this is the level of evidence that is demanded in health technology assessment. Judging the value of treatments in the face of discordant evidence between efficacy and effectiveness studies is an unenviable task for bodies such as NICE.

Opening up the black box to understand how people engage with cCBT
The REEACT benefited from a concurrent qualitative evaluation which has been published in BMJ Open by Knowles and colleagues.4 So whilst many correspondents speculated as to why cCBT worked for some but not for others, we have the benefit of some real data to illuminate this question. The main themes that emerged were the fact that people with depression are often demotivated by this condition and they found it difficult to maintain their enthusiasm for a computer-based treatment modality. Participant experience was on a continuum, with some patients unable or unwilling to accept psychological therapy without interpersonal contact while others appreciated the enhanced anonymity and flexibility of cCBT. The majority of patients were ambivalent, recognizing the potential benefits offered by cCBT but struggling with challenges posed by the severity of their illness, limited support and lack of personalization of programme content. Both positive and ambivalent patients perceived a need for a greater level of monitoring or follow-up to support completion, while negative patients reported deliberate non-adherence due to dissatisfaction with the programme. We also learned that many people valued the regular contact provided via telephone support. We knew that several participants continued to accept support phone calls despite no longer using the computer programmes. This raises the question about what level of support is needed over and above that currently offered in the NHS for cCBT to work. This is an interesting question but is a different question from that which was asked in the REEACT trial.

(B)leading edge technology?
One correspondent pointed out that Beating the Blues and MoodGYM were not now considered to be leading edge computer technologies. We have sympathy with this position, since inherent in computer technology is the need for continual innovation. However we counter this argument with the fact that there was existing trial based evidence that these technologies could work,5 and these were NICE-recommended treatments at the time of design of the REEACT trial.

The lack of engagement and absence of a clinically important effect were so clear that the REEACT trial raises fundamental questions about the way in which people with depression interact with computers or indeed other self-help technologies. This goes beyond the choice of computer programme. We would also assert that it remains important that all new technologies should be trialed since effectiveness can never be assumed. Such technologies are never resource free, and there is an opportunity cost when healthcare systems invest in unevaluated technologies which later turn out to be ineffective.

What is to be done?
We remain optimistic that cCBT does have a role in the management of depression but the REEACT trial shows that the expectations of cCBT do not readily translate into tangible patient benefits under conditions of routine care and delivery. Our main conclusion is that the level of support that needs to be offered in the context of routine care is much higher than is currently the case in publicly funded healthcare systems such as the NHS. Our qualitative research4 and systematic review evidence3 supports this assertion, and suggests that greater levels of guidance and support might result in patient benefit. This is an empirical question which needs to be answered in a well-designed pragmatic effectiveness study with a model of care which exceeds usual levels of NHS support. Fortunately we have undertaken such a study (REEACT2 ISRCTN55310481) and the results of this research will be available in 2016.

Finally we thank the correspondents for an interesting debate, and the interest in our trial reflects the importance of this topic and the need for effective low intensity interventions for depression that can be delivered at scale in the NHS. We also note that our trial has generated a debate about the value of computer-mediated psychological therapy which has not hitherto taken place.

1. Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, et al. Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. BMJ 2015;351:h5627.
2. Schwartz D, Lelloch J. Explanatory and pragmatic attitudes in therapeutic trials. Journal of Chronic Diseases 1967;20:637-48.
3. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: A meta-analysis. Cog Behav Ther 2009;38(4):196-205.
4. Knowles SE, Lovell K, Bower P, Gilbody S, Littlewood E, Lester H. Patient experience of computerised therapy for depression in primary care. BMJ open 2015;5(11):e008581.
5. Proudfoot J, Goldberg DP, Mann A, Everitt B, Marks IM, Gray JA. Computerised, interactive, multimedia cognitive behavioural therapy for anxiety and depression in general practice. Psychol Med 2003;33:217-27.