Jump to comment:

• Approaches for assessing agreement in continuous measurements in a multi-observer setup, comment on “Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre”
  Jens Borgbjerg, Martin Bøgsted and Heidi Søgaard Christensen
  Published on: 14 January 2021

• Published on: 14 January 2021

  Approaches for assessing agreement in continuous measurements in a multi-observer setup, comment on “Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre”

  • Jens Borgbjerg, Consultant radiologist, MD Akershus University Hospital, Norway
  • Other Contributors:
    • Martin Bøgsted, Professor, Senior Biostatistician, PhD
    • Heidi Søgaard Christensen, Post Doc, PhD

  We read with interest the article by Woo and colleagues, evaluating sensitivity of statistical methods for detecting different levels of interobserver variability in CT measurements of cancer lesions [1]. It is increasingly recognized that in order to evaluate the efficacy of medical imaging there is a need to conduct multi-observer studies in which proper statistical analysis is a critical component [2]. Thus, the study by Woo et al. is a welcome addition to the literature, and the authors are commended for providing open access to data.
  The authors supplemented an observed dataset based on the diameters of 10 CT lesions measured by 13 observers by generating two additional datasets of increased and decreased measurement variability, respectively.
  These three datasets were used to compare three statistical approaches 1) intraclass correlation coefficient (ICC), and what the authors refer to as 2) outlier counts (score) from standard Bland-Altman plotting with limits of agreement, and 3) outlier score from Bland-Altman plotting with fixed limits of agreement.

  We have a few comments.

  We ardently agree with the authors that although the widely used ICC accommodates a multi-observer setup, it is not an ideal method for evaluating interobserver variability; the ICC reveals little about the degree of discrepancy nor supply information to investigate whether the variability may change with the magnitude of measurements (e.g. to reveal that the diameter...

  Show More

  We read with interest the article by Woo and colleagues, evaluating sensitivity of statistical methods for detecting different levels of interobserver variability in CT measurements of cancer lesions [1]. It is increasingly recognized that in order to evaluate the efficacy of medical imaging there is a need to conduct multi-observer studies in which proper statistical analysis is a critical component [2]. Thus, the study by Woo et al. is a welcome addition to the literature, and the authors are commended for providing open access to data.
  The authors supplemented an observed dataset based on the diameters of 10 CT lesions measured by 13 observers by generating two additional datasets of increased and decreased measurement variability, respectively.
  These three datasets were used to compare three statistical approaches 1) intraclass correlation coefficient (ICC), and what the authors refer to as 2) outlier counts (score) from standard Bland-Altman plotting with limits of agreement, and 3) outlier score from Bland-Altman plotting with fixed limits of agreement.

  We have a few comments.

  We ardently agree with the authors that although the widely used ICC accommodates a multi-observer setup, it is not an ideal method for evaluating interobserver variability; the ICC reveals little about the degree of discrepancy nor supply information to investigate whether the variability may change with the magnitude of measurements (e.g. to reveal that the diameter of large abdominal aneurysms are less precisely estimated compared to smaller ones). Further, deciding what value constitutes sufficiently high reliability is often made in a subjective fashion. Lastly, since the ICC is heavily dependent on between-subject variation, it may produce a high value simply due to a heterogeneous patient group.

  Regarding statistical approach number two, which references the also widely used Bland-Altman method, it should be stressed that in its original formulation Bland and Altman described a method for assessment of agreement between continuous measurements completed by precisely two methods (or observers) on a number of subjects [3]. The Bland-Altman method entails plotting the difference between paired measurements against the average of the same pair. Furthermore, the approach defines the “limits of agreement” which are calculated as the mean of differences +/- 1.96 times the standard deviations of the differences. When these limits of agreement are deemed small and not of clinical importance, two measurement methods can be considered interchangeable. However, as the limits of agreement are only estimates, confidence intervals should be calculated and reported [4].

  Deviating from the original approach proposed by Bland and Altman, Woo et al. counted outliers of the 95% limits of agreement constructed from all of 78 possible pairs from within the group of the 13 radiologists. The authors observed that the proportion of outliers in the three datasets were the same, and concluded that there was no evidence to support approach number two for detecting different levels of interobserver variability.
  Here it should be mentioned that the 95% limits of agreement is an estimate of the range over which one would expect 95% of differences to lie, and this range will vary in accordance with the variability of observed differences. Hence, based on the generated datasets and the formulation of the limits of agreement it is entirely expected that the proportion of outliers would not differ between the evaluated datasets.

  Regarding statistical approach number three which also refers to the Bland-Altman terminology of limits of agreement, the approach by Woo et al. basically determined the proportion of differences of observer pairs that were less than 20%. As stated by the authors such an approach where proportions of differences are determined based on a given limit has been used previously in a number of studies to evaluate interobserver variability in a multi-observer setup [5]. Given the original formulation of the Bland-Altman method which is limited by only accommodating two observers/methods, we believe that it is confusing to the readership of BMJ and in general to refer to these proportions as “Bland-Altman scores”.

  Nevertheless, an extension of Bland-Altman’s method to provide a straightforward statistical approach for evaluating agreement in a multi-observer setup has been proposed [6]. Jones and colleagues defined the limits of agreement with the mean (LOAM) which is a measure of how much an individual observer’s measurement deviates from the subject-specific mean measurement of all observers. Furthermore, a so-called agreement plot displaying the observed differences against the observed subject-specific average is constructed. The authors of the present letter have recently expanded the method by refining the definition of the LOAM, providing formulas to assess the variation of the LOAM estimate, and extending the method to multiple observations per observer [7]. The method assumes that measurements follow a two-way random effects model, where both observers and subjects are viewed as a random sample from a larger population that we want to make inference about.

  Using this approach, the estimated 95% LOAM for the abovementioned decreased, observed, and increased datasets are ±0.32 cm (95% CI: 0.29 to 0.39 cm), ±0.54 cm (95% CI: 0.48 to 0.64 cm), and ±0.76 cm (95% CI: 0.67 to 0.90 cm), respectively. This demonstrates that the LOAM reveal large differences in the variability of the three datasets. The variance components of the model can be estimated such that the order of magnitude of the inter-observer variation in relation to the intra-observer variation can be compared. In addition, the agreement plot can help to detect whether the spread of the differences is associated with the size of the measurements and whether some observers tend to always make large, small, or more varying measurements.

  Tables with complete LOAM calculations and accompanying agreement plots based on the data provided by Woo et al. can be found at the repository of the R-package loamr for the LOAM method: https://github.com/HaemAalborg/loamr. A direct link to a word file containing the aforementioned analysis can be found here: https://github.com/HaemAalborg/loamr/raw/master/Woo/analysis.docx.

  We recommend using the LOAM statistical method when evaluating agreement between continuous measurements in a multi-observer setup.

  ------------------------

  1. Woo, M., Heo, M., Michael Devane, A., Lowe, S. C. & Gimbel, R. W. Retrospective comparison of approaches to evaluating inter-observer variability in CT tumour measurements in an academic health centre. BMJ Open vol. 10 e040096 (2020).
  2. Farzin, B. et al. Agreement studies in radiology research. Diagn. Interv. Imaging 98, 227–233 (2017).
  3. Bland, J. M., Martin Bland, J. & Altman, D. STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT. The Lancet vol. 327 307–310 (1986).
  4. Shieh, G. The appropriateness of Bland-Altman’s approximate confidence intervals for limits of agreement. BMC Medical Research Methodology vol. 18 (2018).
  5. Jaakkola, P. et al. Interobserver variability in measuring the dimensions of the abdominal aorta: comparison of ultrasound and computed tomography. Eur. J. Vasc. Endovasc. Surg. 12, 230–237 (1996).
  6. Jones, M., Dobson, A. & O’Brian, S. A graphical method for assessing agreement with the mean between multiple observers using continuous measures. Int. J. Epidemiol. 40, 1308–1313 (2011).
  7. Christensen, H. S., Borgbjerg, J., Børty, L. & Bøgsted, M. On Jones et al.’s method for extending Bland-Altman plots to limits of agreement with the mean for multiple observers. doi:10.21203/rs.3.rs-42805/v2.

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top