Changes to pre-specified outcomes must be declared and explained in the published trial report
CCBYOpen access
Rapid response to:
Research
Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial
Changes to pre-specified outcomes must be declared and explained in the published trial report
We thank the authors of the REEACT trial for their response to our letter.
The protocol they cite as “pre-specified” [1] is dated 10th December 2012, however data collection for the trial started in December 2009 - 3 years earlier [2]. We have therefore used the ISRCTN registry entry that predates the start of the trial to establish the pre-specified outcomes.
The authors state that in their ISRCTN registry entry the primary outcome is the PHQ-9 [3]. However, in the ISRCTN registry entry [4], the primary outcome is in fact: “depression severity and symptomatology as measured by a validated self-report measure (the Patient Health Questionnaire [PHQ-9]) and the International Classification of Diseases (ICD-10) depression score at four months”. The only reference on Google to “International Classification of Diseases (ICD-10) depression score" is the REEACT trial’s ISRCTN registry entry. We suspect the phrasing may refer to the WHO’s “Major Depression Inventory”, but believe this constitutes an imprecisely specified outcome. In any case it is clear from the registry entry that there is an additional outcome to the PHQ-9, that has not been reported or mentioned in the BMJ trial publication [2].
Item 6a of the CONSORT statement states that “all outcome measures, whether primary or secondary, should be identified and completely defined” and that details of how and when they were assessed should be reported in the trial publication [5]. Item 6b further states that “any changes to trial outcomes after the trial commenced, with reasons” should also be reported. Where pre-specified outcomes will be reported elsewhere, or non-prespecified outcomes have been added to the published report, this should be declared. Furthermore any changes to the pre-specified primary outcome, even if considered trivial, should be set out in the paper.
The authors go on to argue that the pre-specified secondary outcomes, “Health state utility (EuroQol [EQ5D]) at 4, 12 and 24 months” are not required to be reported as they represent economic as opposed to clinical results. We disagree: publishing quality of life data in a separate academic paper is in no sense a recognised universal convention, and we therefore think that readers should have been told that an extra three secondary outcomes were pre-specified and that there was a plan to report these elsewhere.
The authors lastly suggest that, where studies are funded by NIHR, readers should already know that further information on changes to the pre-specified outcomes could be gleaned by reading various other online NIHR documents associated with the trial. We view that suggestion as problematic. By the authors’ own tacit admission these documents were not available when their BMJ paper was published, and when our assessment was conducted. Furthermore, NIHR-funded trials represent only a tiny fraction of medical research, and so very few readers will know that additional documents may be available, where outcome switching may be declared or discussed, for trials with this specific funding source. CONSORT guidance states clearly that the information needed to detect such changes should be presented clearly in the trial report. From our experience it can take up to 5 hours per trial to check for outcome switching, even using only the protocol, registry entry, and journal publication. Relying on readers to check, even in these documents alone, is therefore impractically onerous. Expecting all readers to know that they should also be reviewing an additional set of documents - which may or may not exist, for a subset of trials depending on funder, with an unspecified time delay - represents an even more impractical method to ensure transparent outcome reporting, especially when the fact that these documents may contain additional information on outcome switching is not even mentioned in the primary academic publication of the trial’s results.
The deviation in sample size mentioned is beyond the aims and protocol of COMPare, and is not mentioned in our assessment or letter. The authors mention that we do not praise other aspects of their trial: again, these are outside the remit of COMPare, which is solely to identify and then flag incomplete outcome reporting to journal editors and the readers of trial reports, in order to drive improvement on this prevalent problem. It may be useful to note that this specific trial scores very much more positively for outcome reporting than most other trials we have reviewed: however this reflects the chaotic state of information management in the medical literature more broadly.
We thank the authors of REEACT for their open discussion of our analysis.
Yours faithfully,
Henry Drysdale, Ben Goldacre, and Carl Heneghan on behalf of the COMPare team.
[2] Gilbody S et al, Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial), BMJ 2015;351:h5627.
[3] Gilbody S et al, REEACT is entirely consistent with CONSORT and the authors exceeded reporting requirements of CONSORT and COMPare, BMJ Rapid Response 21/12/2015.
[5] Moher D et al, CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials, BMJ 2010; 340:c869.
Competing interests:
No competing interests
12 January 2016
Henry M Drysdale
Medical Student
Ben Goldacre, Carl Heneghan
University of Oxford
Centre for Evidence-Based Medicine, Nuffield Department of Primary Care Health Sciences, New Radcliffe House, 2nd floor Radcliffe Observatory Quarter, Woodstock Road , Oxford OX2 6GG
Rapid Response:
Changes to pre-specified outcomes must be declared and explained in the published trial report
We thank the authors of the REEACT trial for their response to our letter.
The protocol they cite as “pre-specified” [1] is dated 10th December 2012, however data collection for the trial started in December 2009 - 3 years earlier [2]. We have therefore used the ISRCTN registry entry that predates the start of the trial to establish the pre-specified outcomes.
The authors state that in their ISRCTN registry entry the primary outcome is the PHQ-9 [3]. However, in the ISRCTN registry entry [4], the primary outcome is in fact: “depression severity and symptomatology as measured by a validated self-report measure (the Patient Health Questionnaire [PHQ-9]) and the International Classification of Diseases (ICD-10) depression score at four months”. The only reference on Google to “International Classification of Diseases (ICD-10) depression score" is the REEACT trial’s ISRCTN registry entry. We suspect the phrasing may refer to the WHO’s “Major Depression Inventory”, but believe this constitutes an imprecisely specified outcome. In any case it is clear from the registry entry that there is an additional outcome to the PHQ-9, that has not been reported or mentioned in the BMJ trial publication [2].
Item 6a of the CONSORT statement states that “all outcome measures, whether primary or secondary, should be identified and completely defined” and that details of how and when they were assessed should be reported in the trial publication [5]. Item 6b further states that “any changes to trial outcomes after the trial commenced, with reasons” should also be reported. Where pre-specified outcomes will be reported elsewhere, or non-prespecified outcomes have been added to the published report, this should be declared. Furthermore any changes to the pre-specified primary outcome, even if considered trivial, should be set out in the paper.
The authors go on to argue that the pre-specified secondary outcomes, “Health state utility (EuroQol [EQ5D]) at 4, 12 and 24 months” are not required to be reported as they represent economic as opposed to clinical results. We disagree: publishing quality of life data in a separate academic paper is in no sense a recognised universal convention, and we therefore think that readers should have been told that an extra three secondary outcomes were pre-specified and that there was a plan to report these elsewhere.
The authors lastly suggest that, where studies are funded by NIHR, readers should already know that further information on changes to the pre-specified outcomes could be gleaned by reading various other online NIHR documents associated with the trial. We view that suggestion as problematic. By the authors’ own tacit admission these documents were not available when their BMJ paper was published, and when our assessment was conducted. Furthermore, NIHR-funded trials represent only a tiny fraction of medical research, and so very few readers will know that additional documents may be available, where outcome switching may be declared or discussed, for trials with this specific funding source. CONSORT guidance states clearly that the information needed to detect such changes should be presented clearly in the trial report. From our experience it can take up to 5 hours per trial to check for outcome switching, even using only the protocol, registry entry, and journal publication. Relying on readers to check, even in these documents alone, is therefore impractically onerous. Expecting all readers to know that they should also be reviewing an additional set of documents - which may or may not exist, for a subset of trials depending on funder, with an unspecified time delay - represents an even more impractical method to ensure transparent outcome reporting, especially when the fact that these documents may contain additional information on outcome switching is not even mentioned in the primary academic publication of the trial’s results.
The deviation in sample size mentioned is beyond the aims and protocol of COMPare, and is not mentioned in our assessment or letter. The authors mention that we do not praise other aspects of their trial: again, these are outside the remit of COMPare, which is solely to identify and then flag incomplete outcome reporting to journal editors and the readers of trial reports, in order to drive improvement on this prevalent problem. It may be useful to note that this specific trial scores very much more positively for outcome reporting than most other trials we have reviewed: however this reflects the chaotic state of information management in the medical literature more broadly.
We thank the authors of REEACT for their open discussion of our analysis.
Yours faithfully,
Henry Drysdale, Ben Goldacre, and Carl Heneghan on behalf of the COMPare team.
References:
[1] REEACT Trial Protocol version 12, 10/12/2012: http://www.nets.nihr.ac.uk/__data/assets/pdf_file/0003/51438/PRO-06-43-0...
[2] Gilbody S et al, Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial), BMJ 2015;351:h5627.
[3] Gilbody S et al, REEACT is entirely consistent with CONSORT and the authors exceeded reporting requirements of CONSORT and COMPare, BMJ Rapid Response 21/12/2015.
[4] REEACT trial registry entry: http://www.controlled-trials.com/ISRCTN91947481/91947481?link_type=ISRCT...
[5] Moher D et al, CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials, BMJ 2010; 340:c869.
Competing interests: No competing interests