Article Text
PDF
Abstract
Introduction Many studies have indicated the impact of bias in dissemination and publication in medical research. Existence of such bias among clinical trials has been repeatedly pointed out, but it has not been well studied in the field of systematic reviews (SRs). We therefore aim to investigate whether or not time lag bias and publication bias in SRs based on statistical significance in results exist. In addition, we will examine at what stage of paper publication process such bias, if any, creeps in.
Method and analysis The present study is a meta-epidemiological study. We will include all SRs of interventions registered in the international prospective register of SRs (PROSPERO) before December 2014 if the SR has completed its analysis irrespective of its publication status. All contact authors of eligible SRs will be asked to participate in a survey administered through the Internet. Our primary outcome is time from protocol registration to full publication of SR as a journal article, defined as time from the registration date to the acceptance date among all the relevant SRs. We will examine the impact of statistically significant findings on the primary outcomes through time to event analyses.
Ethics and dissemination Ethics approval will be obtained from the Ethical Committee of the Kyoto University Graduate School of Medicine. This protocol has been registered in the University Hospital Medical Information Network Clinical Trials Registry. We will publish our findings in a peer-reviewed journal and also may present them at conferences. Trial registration number: UMIN000028325
- publication bias
- time lag bias
- systematic reviews
- protocol registration
- meta-epidemiological study
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first study to contact the authors of unpublished systematic reviews (SRs) and investigate the existence and the magnitude of time lag bias in the realm of SRs.
The factors associated with time to publication of SRs will inform potential preventive measures for these biases.
The generalisability will be limited because the analyses investigating the biases include only those who registered the protocol in the PROSPERO and responded to our survey.
The time to publication may not reflect true time lag bias that is the period between the initiation of the SR and its publication because protocols may be registered after their analyses
Introduction
Many studies have reported bias in dissemination and publication of research findings in medicine.1–5 Bias in dissemination and publication can be introduced at all stages of the publication process after study commencement, such as conducting research and writing up of manuscripts by investigators, and acceptance by journal editors or peer reviewers.
Publication bias occurs when the authors’ decision to write and submit the results or editors’ acceptance for publication is influenced by the direction or strength of the study findings.6 When this happens, findings of published studies will be systematically different from those of unpublished studies and hence from the underlying truth (figure 1).7
Concept of the present study. Study with statistically significant findings may be more likely to be submitted, accepted and published (publication bias) and published earlier (time lag bias).
Time lag bias is one aspect of publication bias which arises when the speed of publication depends on the direction and nature of the results.2 7 8 In randomised clinical trials (RCTs), trials without statistically significant results take a longer time before publication than trials with statistically significant results (figure 1).4 Prior studies found that publication bias is often due to investigators’ failure to write up and submit rather than due to editors or reviewers.7 9 With respect to time lag in particular, one can characterise the phenomenon as either delay due to non-significant results, or expedited submission and publication due to significant results.
Bias in dissemination and publication among clinical trials has been well studied; however, it has not been much studied in the field of non-Cochrane systematic reviews (SRs).10 A study suggested that statistically significant results were not associated with time to publication of Cochrane reviews.11 This result may not apply non-Cochrane reviews because several studies suggested that there was deference in the quality of reporting between Cochrane and non-Cochrane reviews.12 13 A survey among first or corresponding authors of SRs indicated that unpublished SRs exist.14 The authors reported common reasons for not publishing SRs including lack of time and the manuscript being rejected by journals. Statistical significance was not reported as being a major barrier or reason for not publishing, but 65% of respondents reported significant results as a significant facilitator for publishing SRs—in other words, authors are more likely, faced with other pressures, to take the time to complete and submit their review when they have positive results to report. Moreover, non-Cochrane SRs were likely to report statistically significant findings and positive conclusions.13 We recently reported that more than 30% of non-Cochrane SRs registered in the international prospective register of SRs (PROSPERO) were not published after at least 50 months of registration.15 These results indicate that time lag bias and publication bias among non-Cochrane SRs may well exist.
This study therefore aims to investigate whether or not time lag bias and publication bias in SRs based on statistical significance exist. We will also evaluate other factors associated with time to publication.
Objectives
We aim to evaluate the association between statistical significance of meta-analysis result and publication status using a number of criteria (full publication in a journal article, submission to any journals, presentation of an abstract at a meeting) among SRs. We also aim to examine other factors of possible influence and publication of SRs.
Methods and analysis
Inclusion and exclusion criteria
All SR protocols of interventions registered in the PROSPERO by 31 December 2014 will be eligible. We have chosen this time limit because we expect that it may often take 3 years to complete and publish SR after its registration.15 We will exclude SRs that include studies other than RCT. We will exclude SRs whose analysis has not been completed and SRs without quantitative synthesis. We will exclude Cochrane protocols and reviews because their publication process is different from general peer-reviewed journals. We will also exclude SRs of diagnostic test accuracy and prognosis, and SRs with network meta-analysis, individual patient data meta-analyses because different factors are likely to be at play. figure 2 shows the expected flow diagram of this study.
Expected flow diagram of the present study. RCT, randomised clinical trials; SRs, systematic reviews.
Search method
We will search the relevant SRs in the PROSPERO. The planned start date of the search will be on 15 November 2017. We will use search filters of ‘Exclude Cochrane protocols’ for type of protocol and ‘Intervention, Prevention or Service Delivery’ for type and method of the review. For protocols with ‘Ongoing and Completed’ stage of review, we will search MEDLINE via PubMed and Google Scholar to find a full publication, using the authors’ names and the keywords for participants or intervention in the PROSPERO because our pilot search indicated the status of the PROSPERO may often not be updated promptly to reflect the true status. For protocols with ‘Abandoned’ stage of review, we will exclude them when their data analyses have not been completed.
Study selection
The selected studies based on a random sample of 500 of the PROSPERO records identified by the initial search will be divided into two subsamples of 250, and two pairs of assessors (YTsuj-YTsut and HT-YK) will assess the eligibility for each set separately. We will resolve disagreements by discussion between the authors, with another author (TAF) acting as an arbiter.
Data extraction
All records of included SR protocols will be downloaded from the PROSPERO for the use of data extraction. Four authors (YTsuj, YTsut, DP and YK) will independently extract the following data from the relevant SRs in the PROSPERO: Registration and anticipated date of completion, the number of authors, funding sources, conflicts of interests (COIs), stage of review, year of registration and countries where the study has been conducted. We will define the primary review question of the SR in terms of participant, intervention, comparator and primary outcome. As the review may provide several comparisons for the same class of interventions and comparators, we will define the primary comparison as the intervention and comparator that are described as the primary or the first one in the intervention and comparator section in the PROSPERO record, and the primary outcomes as all outcomes listed in the primary outcome section of PROSPERO. If the primary outcome was analysed for multiple time points, we will use the meta-analysis result that included the largest number of studies. Should the primary outcome be missing in the primary outcome section of PROSPERO, we will define the one mentioned first as the primary outcome. For SRs published in a journal article, we will extract the date of acceptance. We choose acceptance date rather than publication date because the interval between acceptance date and publication date depends on many external factors unrelated to publication bias. If acceptance date is not available, we will use the publication date for SRs published in open access journals, and the date of online publication ahead of print for SRs published in other journals. We will contact the authors or the editorial office if the relevant date is missing.
Survey
For potentially eligible SRs, we will contact the authors listed in the ‘Contact details for further information’ of the PROSPERO records and will ask them to respond to a survey through the internet. Through the survey, all contact authors of potentially eligible unpublished SRs will be asked whether or not the SR analysis has been completed. Additionally, all contact authors of published or completed but not published SRs will be asked the following information: whether or not each of the primary outcomes in the primary comparison (as defined by a decision rule described in Data extraction section above) was statistically significant; the review team’s involvement in any of the trials included in the SR; the author’s experience to publish an SR as a lead author before the PROPERO registration; relationship with a private for-profit consulting firm for SRs; and main barriers to publish the SR according to a classification used in a previous study.14 In addition, we will ask if someone has published a review addressing the same question. If the authors have completed the analysis but not published it, the following information will be sought: the number of included trials in the SR; whether or not they have presented the SR at a scientific conference; and whether or not they have submitted the SR to any journals. All surveys will be administered using Google Form (https://www.google.com/intl/en/forms/about). We will send a reminder along with the Google form link 1 week after the initial contact. If the authors have not responded by that time, we will repeat this process twice.
Sample size
Based on previous findings, we estimate that proportion of statistical significance is approximately 50% among SRs registered in the PROSPERO, and the median time to publication are 15 months for SRs with statistical significance and 25 months for SRs without statistical significance.7 13 15 16 A total of 110 events is expected to provide approximately 90% power to detect the difference with an assumed type I error of 0.05 (two-sided). Assuming that approximately 50% of the eligible protocols are not published, we would need a total of 220 SRs. We aim to repeat the random sampling of relevant SRs from PROSPERO until the number of eligible SRs whose authors respond to our survey reaches 220, or the registry is exhausted. The sample size will be modified when the actual proportion of unpublished reviews among the eligible protocols in our first batch is less than 30% or more than 70%.
Data analysis
Primary analyses
Our primary outcome is (1) time from protocol registration to publication of SRs in journal articles, defined as time (months) from the registration date to the acceptance date. We will draw Kaplan-Meier curves for time to publication, classified by SRs with and without statistically significant meta-analysis results. We will examine the association of statistical significance and time to publication using Log-rank test. We will then use Cox proportional hazard model for a multivariable analysis to adjust for two apparent confounders, namely the number of included studies and year of registration. The number of included studies possibly associates with the importance of the topic, and the increase of statistical power. The year of registration may associate with the effect size of the intervention, and the acceptance rate of SRs.
Secondary analyses
Secondary outcomes will include (2) proportion of SR published in journal articles, (3) a composite outcome of full publication or presentation at scientific conferences and (4) submission to any journals. We will describe a table showing the proportion of statistical significance and summarise the characteristics of included SRs classified by full publication, submission, presentation and no dissemination. We will analyse the association between statistically significant findings and the secondary outcomes using univariable logistic regression, and multivariable logistic regression to adjust for the number of included studies and year of registration.
When there is a statistically significant association between statistically significant findings and time to publication, we will explore the predictors of time to publication other than statistical significance, number of included studies and year of publication. First, we will examine the association of the proportion or time to publication with academic or financial COI, experience of SR publication, country of contact author’s affiliation (English speaking or not) or multinational collaboration using Log-rank test. Then, we will use Cox proportional hazard model to explore the influence of these factors on the association between statistical significance and time to publication. We will summarise proportion of SRs that have not reported the primary outcomes as defined in the protocol among published SRs. We will describe whether or not the PROSPERO status reflect the true publication status and will summarise the barriers for completed SRs to publish.
Continuous variables will be shown as mean (standardised deviation) and categorical variables will be expressed as numbers with percentage (%). A two-sided p value smaller than 0.05 will be considered statistical significance. We will use Stata/SE, V.14.0 (StataCorp, College Station, TexasX, USA) for all analyses.
Sensitivity analysis
We will conduct the following prespecified sensitivity analysis: Restricting SRs to those in which the authors have clearly predefined primary outcomes including the time point of measurement in their protocols
Ethics and dissemination
Ethics approval will be obtained from the Ethical Committee of the Kyoto University Graduate School of Medicine. This protocol has been registered in the University Hospital Medical Information Network (UMIN) Clinical Trials Registry (Trial registration number: UMIN000028325). The planned completion date of the present study is 31 December 2018. We will publish our findings in a peer-reviewed journal and also may present them at conferences.
Discussion
SRs with adequate quality have potentials to alter the daily clinical practice, and are useful resources in developing clinical practice guidelines and policies. Time lag bias and publication bias can be a strong barrier to research transparency and integrity. Before the launch of PROSPERO, it was difficult to find SR protocols that remained unpublished. After 6 years from its launch, the registry will enable us to evaluate publication bias and time lag bias in the SR field, as was the case of clinical trial registries.7 This is the first study to contact the authors of unpublished SRs and investigate the existence and the magnitude of these biases. We recently suggested that protocol registration was not associated with reporting of statistical significance.16 The factors associated with publication or time to publication of SRs will inform potential preventive measures for these biases. We also aim to describe the publication rate of registered protocols and the proportion of published SRs that have not reported primary outcomes as defined in the protocol, and check whether the PROSPERO status reflects the true publication status.
There are several expected limitations for this study. First, the generalisability will be limited because the analyses investigating the biases include only those who registered the protocol in the PROSPERO and respond to our survey. In order to increase the response rate, we will remind the contact authors up to two times for every week if they do not respond to our survey. Second, the time to publication may not reflect true time lag bias that is the period between the initiation of the SR and its publication because protocols may be registered after their analyses. The PROSPERO prohibits registration of completed reviews but some may nonetheless register after the completion of the analysis.17 Because SRs that registered after their completion are more likely to have statistically significant findings and may be published earlier, if there are many such protocols, our hypothesised association may be overestimated. Third, there can be unmeasurable confounding such as the methodological quality of the protocol. We will therefore perform a sensitivity analysis to restrict studies that adequately predefined their primary outcomes. Fourth, the accuracy of PROSPERO filters is unknown, but the use of these filters is not likely to bias the results of the present study. Finally, unlike clinical trials, the authors may not intend to find statistical significance in the realm of SRs. This may bias the association between statistical significance and publication towards null.
In conclusion, this study will provide comprehensive investigation about time lag bias and publication bias in the realm of SRs using the first global registry for SRs.18 The expected findings will show the needs and the key factors to prevent such biases.
Acknowledgments
We would like to thank Center for Reviews and Dissemination at University York, UK, for managing the PROSPERO.
References
Footnotes
Contributors YTsuj and YTsut contributed equally to this work. YTsuj, YTsut, HT, YK, YY, DP, GHG, SF and TAF contributed to the conception and design of the research. YTsuj and TAF are fully responsible for writing the protocol. TAF supervised the research, and all authors gave final approval of the protocol before submission. After the publication of the protocol, we plan the following contributions of each author: YTsuj, Ytsut, HT, YK and DP will screen the relevant records of the PROSPERO, and extract data. YTsuj, Ytsut, HT, YK, DP, GHG and TAF will contact the authors for additional information. YTsuj, Ytsut, YY and TAF will conduct the data analysis without blinding of the data. YTsuj, Ytsut and TAF will write the manuscript. GHG, SF and TAF will revise the manuscript critically for important intellectual content. TAF will supervise the research.
Funding This study is supported in part by JSPS KAKENHI (Grant-in-Aid for Scientific Research) Grant Number 17K19808 to TAF. The funder plays no role in developing the protocol.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.