Article Text

Protocol
Quality and completeness of, and spin in reporting of, pilot and feasibility studies in hip and knee arthroplasty: a protocol for a methodological survey
  1. Zinnia Chung1,
  2. Lawrence Mbuagbaw2,3,
  3. Lipalo Mokete4,
  4. Mohit Bhandari5,
  5. Lehana Thabane6,7
  1. 1Michael G. DeGroote School of Medicine, McMaster University, Hamilton, Ontario, Canada
  2. 2Department of Health Research Methods Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
  3. 3Biostatistics Unit, Father Sean O'Sullivan Research Centre, St Joseph's Healthcare Hamilton, Hamilton, Ontario, Canada
  4. 4Division of Orthopaedic Surgery, University of the Witwatersrand Johannesburg, Johannesburg, Gauteng, South Africa
  5. 5Department of Surgery, McMaster University, Hamilton, Ontario, Canada
  6. 6Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
  7. 7Research Institute, St Joseph's Healthcare Hamilton, Hamilton, Ontario, Canada
  1. Correspondence to Dr Lehana Thabane; thabanl{at}mcmaster.ca

Abstract

Introduction Pilot or feasibility trials examine the feasibility, viability and recruitment potential of larger, main trials. Specifically, a pilot trial can be instrumental in identifying methodological issues essential to the development of an effective research protocol. However, numerous studies published as pilot or feasibility studies have demonstrated notable inconsistencies in the nature of information reported, resulting in poor-quality and incomplete reporting. It is unclear whether such low quality or incompleteness of reporting is also prevalent in arthroplasty pilot trials.

Methods and analysis This protocol outlines a methodological survey examining the completeness of reporting among hip and knee arthroplasty pilot trials in accordance with the Consolidated Standards of Reporting Trials (CONSORT) 2010 extension to pilot trials. Secondary objectives include: (1) determining the prevalence of ‘spin’ practices, defined as: (a) placing a focus on statistical significance rather than feasibility, (b) presenting results that show the trial to be non-feasible as feasible or (c) emphasising the effectiveness or potential intervention benefits rather than feasibility; (2) determining factors associated with incomplete reporting, and ‘spin’. A search of PubMed will be conducted for pilot trials in hip or knee arthroplasty published between 01 January 2017 and 31 December 2023. Following screening, appropriate data will be extracted from eligible publications and reported as descriptive statistics, encompassing elements of the CONSORT checklist associated with completeness of reporting. Logistic regression analysis and Poisson regression will be used to analyse factors associated with completeness of reporting and spin.

Ethics and dissemination This methodological review does not require formal ethical approval, as it will solely involve the use of published and publicly reported literature. The results of this study will be disseminated through submission to peer-reviewed journals and academic conference presentations. Study details will be sent to McMaster University’s media coordinators to be shared through the institution’s research-focused platforms.

  • feasibility studies
  • statistics & research methods
  • hip
  • knee
  • orthopaedic & trauma surgery
  • orthopaedic & trauma surgery
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

  • This protocol involves the appraisal of reporting completeness and quality through three different checklist perspectives; the Consolidated Standards of Reporting Trials 2010 checklist extension, a modified spin reporting framework and a shortened key feasibility items checklist.

  • The use of stratified random sampling will allow for studies from all included publication years to be equally represented within the analysed population.

  • As a methodological survey, this study encompasses a smaller sample size with limited amount of studies from each investigated year.

  • The articles included for analysis will be restricted to works that have been published in English and made available on the PubMed database from 2017 to 2023.

  • This study will focus directly on hip and knee arthroplasty procedures; the results may not be generalisable to other forms of arthroplasty and their patient populations.

Introduction

Pilot studies and feasibility studies examine the feasibility, viability and recruitment potential of large main trials.1 Often conducted on a smaller scale, these studies offer insights that contribute towards enhancing the quality, validity and probability of success of subsequent main studies.1 Specifically, the methodological issues raised by pilot and feasibility studies aid in informing the development of an effective research protocol.2 Here, they can highlight considerations related to logistical elements such as measurement tools, sample sizes and parameters. However, a notable challenge arises from the inconsistencies in reporting found among studies published under the label of ‘pilot’ or ‘feasibility’.3 Such concerns have been associated with a historical lack of reporting guidelines for pilot and feasibility studies, resulting in unclear and wide-ranging objectives.4 Kaur et al’s review of pilot studies noted that only 58% of entries stated their specific purposes for piloting, with 12% progressing to a definitive trial.5 Consequently, these varying viewpoints have been reflected in the resulting body of the literature, where studies classified under ‘pilot’ may be conducted with different foci, purposes and objectives. In these cases, ambiguous objectives, minimal methodological focus and varied reporting standards result in a wide range of publications that lack standardisation.

Previous research has identified discrepancies in pilot and feasibility study reporting.3 Systematic reviews have identified the need for improved standardisation among reported trials to facilitate proper evaluations and analyses of field developments.6 A 2010 review additionally revealed that 81% of intervention-based pilot studies had insufficient sample sizes.7 8 In response to these growing weaknesses, Eldridge et al. proposed an extension to the 2010 Consolidated Standards of Reporting Trials (CONSORT) statement.1 9 This 26-item checklist extension was built on the original CONSORT guidelines, which were designed to improve the reporting quality found in randomised controlled trials (RCTs).10 Published in 2016, the CONSORT extension applies to any randomised study conducted before an RCT (including pilot, feasibility trials and studies) and aims to enhance reporting quality across disciplines.1 Despite widespread acceptance, implementation varies greatly across different fields, as demonstrated by McGrath et al’s documentation of suboptimal reporting practices in paediatric urology-based pilot studies.11–13

In addition to content variations, pilot and feasibility studies may also incorporate reporting approaches that are misrepresentative of primary outcomes of feasibility, termed ‘spin’. These practices divert attention from non-feasible methodologies by emphasising factors such as statistical significance or efficacy.14 By selectively emphasising certain study findings, researchers can present non-feasible methodology through a favourable lens, misconstruing a reader’s interpretation of the presented information. While the prevalence of ‘spin’ practices has been investigated in areas of published biomedical literature (encompassing clinical trials, observational studies, meta-analyses, systematic reviews and diagnostic accuracy studies), its impact in pilot trials remains unexplored.15 Collectively, gaps in content details and misleading communication may detract from the reliability of pilot and feasibility trials, similar to what has been found in main trials—emphasising the importance of investigating ‘spin’ in the practice of pilot trials.16

This study specifically focuses on pilot trials in total hip and knee arthroplasty, which remain among the most commonly performed joint replacement procedures.17 As consistent and cost-effective options for patients experiencing osteoarthritis, inflammatory arthritis or joint disorders, these procedures stand among the top three inpatient surgeries performed in Canada annually.18 Similarly, trend projections from Otten et al from the Netherlands further suggest that the number of arthroplasties may increase by 149% and 297% for hip and knee procedures by 2030, respectively.19 Given the sustained demand for arthroplasties, ongoing research developments are vital for maintaining quality and improving patient outcomes. Specifically, as of January 2024, clinicaltrials.gov reports 308 and 75 actively recruiting clinical trials in hip and knee arthroplasties, respectively. Therefore, to ensure that studies can keep pace with the field’s growth, proper reporting practices and optimal use of pilot studies will continue to play essential roles. With this as the case, the primary objective of the study is to evaluate the completeness of reporting among hip and knee arthroplasty pilot and feasibility trials, as judged by adherence to the CONSORT extension checklist for pilot trials (see online supplemental appendix A). In the process, secondary aims are: (1) to identify the prevalence of ‘spin’ reporting techniques (defined using a modified adaptation of the original ‘spin’ criteria identified by Boutron et al16 see Outcome definitions section), (2) to evaluate the completeness of reporting in relation to key feasibility items (derived from the complete extension checklist; see online supplemental appendix B) and (3) to identify factors associated with the completeness of reporting based on the CONSORT extension.

Methods and analysis

Planned start date: 01 May 2024.

Anticipated end date: 01 December 2024.

Patient and public involvement

None.

Eligibility criteria

Pilot and feasibility studies must meet the following criteria for inclusion:

  1. Study type: described as a pilot study or a feasibility study, with ‘study’ and ‘trial’ used interchangeably.

  2. Publication date: published between 1 January 2017 and 31 December 2023 in the PubMed database (via OVID).

  3. Topic: focus on hip or knee arthroplasty procedures performed in humans, specifically investigating interventions intended to improve preoperative, procedural or clinical outcomes.

  4. Language: published in the English language.

Identification of pilot and feasibility studies

Pilot and feasibility studies will be identified based on the inclusion of the terms ‘pilot’ or ‘feasibility’ in the publication title. Publications that do not meet these criteria will be further screened based on their abstracts and methodology to determine eligibility based on the study definitions published by Eldridge et al.4

Exclusion criteria

  1. Non-pilot RCTs in hip or knee arthroplasty.

  2. Studies performed on or including an animal population.

  3. Studies that are unpublished or pending publication.

  4. Studies published before or after 01 January 2017 to 31 December 2023.

  5. Secondary research articles (review articles, meta-analyses).

  6. Case reports.

  7. Study protocols.

Search strategy

A search of PubMed will be conducted using the search strategy documented in online supplemental appendix C to identify relevant publications for inclusion. Medical subject headings will be used alongside keywords to identify relevant pilot or feasibility trials in hip and knee arthroplasty.

Data extraction and synthesis

Search results will be exported to Microsoft Excel, where duplicate entries will be removed. All entries will subsequently be screened by title in accordance with the inclusion and exclusion criteria by a primary reviewer. Following full-text screening, a sample of 20 entries will be screened by a second reviewer to ensure consistent interpretation of concepts and accuracy in determining study eligibility.

Following the screening phase, data extraction will be completed independently by the primary reviewer, with a sample of 20 entries again being extracted by a second reviewer to check for accuracy and quality. Any discrepancies between the two reviewers will be resolved by the intervention of a third reviewer.

Data extraction

The following information will be extracted from the included publications:

  1. Date of publication.

  2. Location of pilot trial identification (whether a publication was classified as a pilot or feasibility study based on the title, abstract or methodology).

  3. Journal impact factor.

  4. Study location.

  5. Study design.

  6. Study population demographics; mean age, age range.

  7. Sample size used.

  8. Primary objectives.

  9. Intervention.

  10. Source of funding.

  11. Adherence to the 2010 CONSORT extension.

  12. Presence of ‘spin’ reporting practices (as outlined in Outcome definitions section).

  13. Adherence to the key feasibility items checklist.

  14. Outcome reported from the pilot trial.

  15. Follow-up actions → whether the full-scale RCT was conducted.

Outcome definitions

  1. Completeness of reporting: for the purposes of this study, the definitions identified by Eldridge et al will be used to define and establish the relationship between feasibility and pilot trials.4 This framework views pilot studies to be a subset of a wider category of feasibility studies, with the latter aiming to ask ‘whether something can be done, should we proceed with it, and if so, how’.4 Comparatively, a pilot study asks ‘the same questions but also has a specific design feature: in a pilot study a future study, or part of a future study, is conducted on a smaller scale’.4 As the CONSORT 2010 extension was developed to encompass both, this study will also include pilot and feasibility studies within the same sample population.10

  2. ‘Spin’: ‘spin’ practices are defined in this study based on the presence of three main categories derived from Boutron et al.16

    • A focus on statistical significance of health or therapeutic outcome(s), rather than feasibility (eg, secondary outcomes).

    • Presenting non-feasible results (statistically non-significant) as feasible or effective.

    • Emphasising effectiveness or potential intervention benefits rather than feasibility.

  3. Key feasibility item checklist: the key feasibility item checklist, also titled the ‘triage checklist’, focuses on a select group of core criteria that should be applied to the reporting of pilot and feasibility studies. Derived from the 2010 CONSORT extension for pilot trials, this shortened checklist establishes a fundamental baseline for researchers to start with. See online supplemental appendix B for the item checklist.

Sampling strategy

Sample size calculation

The required sample size for this study will be determined using the approach outlined by Isiguzo et al.20 This method is based on a 95% CI and involves the following calculation:

n=1.962 (P0 (1−P0)/E2) where

P0=prior estimate of the proportion of studies with adequate reporting.

E=target margin of error.

Based on previous studies, adherence to CONSORT items tends to range from 0.25 to 0.70.21 22 Online supplemental table 1 provides estimates of the sample sizes appropriate for P0 values within this range based on a 95% CI. Within this identified range, we have calculated the value of 147 to be the minimum acceptable population size for this study (determined using a P0 of 0.25 and with a margin of error of 0.05).

Sampling strategy

After completing the database search, articles will be ordered for full-text screening. Following the identification of all eligible studies, articles will be arranged by year and a minimum random sample of 21 studies will ideally be selected from each of the 7 following years (2017–2023) to comprise the full study minimum population of 147. This approach is designed to ensure a holistic view of CONSORT extension implementation across the years since its publication in 2016.

Statistical analysis

Primary outcome measures

To assess the quality of reporting, the CONSORT 2010 extension for pilot trials will be used. Items that are reported as present will be indicated by ‘1’. Unreported checklist items will be assigned as ‘0’. Elements that are not present in the evaluated publications will be labelled as N/A. Completeness of reporting will be represented as a count representing the number of reported items, among those applicable (not marked ‘N/A’). We will also report the percentage of adequate reporting defined as reporting at least 75% of applicable checklist items.

Secondary outcome measures

The first secondary outcome will be ‘spin’ reported as a composite outcome of whether any of the three instances of ‘spin’ occurred. We will also report the frequency or prevalence of each of the components of ‘spin’. The second will be a count of reported items based on the shortened triage checklist of key feasibility items (see Outcome definitions section).

Method of analysis

CONSORT 2010 extension

Completeness of reporting will be assessed in accordance with adherence to the CONSORT 2010 extension to pilot trials checklist. We will use descriptive statistics to represent the average number of items reported, along with the number and percentage of studies that include each item.

‘Spin’ practices

We will use descriptive statistics to report an estimate of the percentage of studies that have at least one of the item definitions of ‘spin’, and the percentage of studies having each item definition with a 95% CI.

Key feasibility items

Completeness of reporting will also be assessed using the key feasibility items (‘triage’) checklist. We will use descriptive statistics to represent the average number of key feasibility items reported, along with the number and percentage of studies that include each key feasibility item with a 95% CI.

Evaluating associated factors associated with key outcomes: number of reported CONSORT items and number of key feasibility outcomes reported

Poisson regression will be used to determine factors that may be associated with the completeness of reporting based on the CONSORT extension checklist and the key feasibility items. The following factors will be explored in these analyses:

  1. Whether the journal endorses the CONSORT statement.

  2. Journal policy requiring the inclusion of the CONSORT checklist during the submission of a manuscript.

  3. The presence of a structured abstract.

  4. Type of intervention.

  5. Source of funding.

These factors have been evaluated in similar studies before, with reported associations with the completeness of reporting.20 23 24 We will examine residuals to assess model assumptions and consider using negative binomial distribution to analyse the data if there is evidence of overdispersion. The results will be reported as incidence rate ratio, corresponding to a 95% CI and associated p values. All p values will be reported to three decimal places with those less than 0.001 reported as p<0.001. The criterion for statistical significance will be set at alpha=0.05 and will not be adjusted for multiple testing since these analyses are exploratory. All analyses will be performed using SAS V.9.4. Please see table 1 for a summary of the study objectives, corresponding outcomes, explanatory variables and method of analysis.

Table 1

Summary of the objectives, corresponding outcomes, explanatory variables and method of analysis

Ethics and dissemination

This methodological review does not require ethical approval. In accordance with TCPS 2, articles 2.2–2.4, the study will solely involve the use of published, peer reviewed and publicly reported literature. No identifiers linking to any individuals will be included.

Dissemination

Results of this study will be disseminated through submission to peer-reviewed journals and academic conferences for presentation. Key information will additionally be sent to McMaster University’s social media coordinators to be shared through the institution’s research-focused platforms.

Ethics statements

Patient consent for publication

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors LT and ZC conceived the study and drafted the manuscript. This included the development of all sections, from the writing of the introduction, formulation of methods and creation of relevant supplemental material (tables and calculations). All authors (LMb, LMo, MB, ZC and LT) reviewed different versions of the manuscript, proposed critical revisions and approved the final version. LT is the guarantor of the study.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.