Recommendations for musculoskeletal ultrasonography by rheumatologists: Setting global standards for best practice by expert consensus

Study design.

We employed a 4-stage consensus defining approach utilizing the Delphi technique (20) and a panel of international experts in MUS (Figure 1). This process involved an initial stage of identification of areas considered to be important for rheumatologists; two iterative rounds of questionnaire completion, feedback, and reflection; and a final stage of analysis.

Delphi process.

The Delphi process is a research technique originally developed in the 1950s by the Rand Corporation and takes its name from the Delphic oracle's skills of interpretation and foresight (20). It is a structured multistage process involving a series of iterative rounds and aims to combine expert opinion to establish a group consensus on a given subject. It can be usefully applied to areas of controversy or areas in which published data are inadequate and involves a series of questionnaires interspersed with controlled feedback (21-23). The Delphi approach was chosen because it represents an established method of obtaining considered opinion of experts, determining levels of agreement, and evaluating the degree of consensus. It has been widely used in various areas of health care research, including the development of guidelines for best practice, content of curricula, and in the definition of professional roles and clinical protocols (24, 25). There are a number of specific advantages to this methodology that were useful for our purposes. It allows a large number of experts to be sampled without geographic constraints; it is relatively inexpensive and straightforward to manage with quantitative and qualitative data being collected via a self-administered questionnaire; and anonymity between panelists avoids dominance of opinions from certain individuals. This overcomes a potential problem of group dynamics that can occur with decision-making committees and encourages true opinion with less pressure to conform to the collective viewpoint of the committee, profession, or establishment. The regular participation in the iterative rounds and controlled feedback allows individuals to change or modify their opinions and encourages panel members to become more involved and motivated to participate in the development of the project—this enhances ownership and improves the likelihood of acceptance of the findings. Potential disadvantages include a lack of data regarding the minimum sample size; difficulties of ensuring representative sampling of experts; no formal definition as to what actually constitutes a consensus; potential reduced response rate in later rounds; and possible lack of accountability because of anonymity within the group.

Identification of the expert panel.

The first stage involved selection of an appropriate panel of experts in MUS. These experts were chosen by Medline literature review to identify authors of relevant peer-reviewed publications; identification of members of the teaching faculty of established MUS training courses; and consultation with committee members from the European League Against Rheumatism, British Society of Skeletal Radiologists MUS working groups, and the Musculoskeletal Ultrasound Society to identify individuals regarded as experts by their peers (17). This group of informed experienced professionals, each with a track record of ongoing practice, research, and teaching, consisted of international experts in the field who were selected as a heterogeneous but representative sample of specialist practitioners in MUS.

Questionnaire design.

The construction of an appropriate data-collection instrument involved a number of steps. A Medline database literature review was conducted to identify areas in which MUS had been utilized and that were relevant to rheumatologic practice. Data from a preceding study, which identified the current MUS practice of the same group of experts, was reviewed (17). In addition, local experts were interviewed and asked to list areas that they perceived to be important. This information was collated and analyzed to produce a 4-page questionnaire that was divided into 4 sections: 1) indications for which a rheumatologist could perform a MUS examination (13 categories); 2) anatomic areas in which it would be appropriate for rheumatologists to undertake MUS (10 categories); 3) the knowledge and skills that a rheumatologist should acquire as part of their training in MUS (14 categories); and 4) a blank section in which the experts were asked to detail any areas not listed that they thought were important together with any other opinions or comments. Panelists were asked to rate their level of agreement or disagreement for each category according to a Likert scale (see below).

A small local pilot study was then undertaken to assess the function of the questionnaire to exclude any difficulties with comprehension or wording, to review its layout, and to assess the feasibility of administration. This resulted in minor modifications. This completed stage 1 of the Delphi process.

The finalized questionnaire was then distributed to our entire expert panel by postal mail as part of stage 2 of the Delphi process. After 4, 7, and 10 weeks, written, e-mail, and personal telephone reminders were made to the nonresponders.

Stage 3 involved summarizing the ratings given by the respondents in the previous round and including these in a further version of the stage-1 questionnaire. Following each question, the percentage distribution of the group responses was listed together with a reminder of the respondent's own previous score. This feedback information was presented to each expert who was asked to study the group replies and indicate whether their individual opinion remained unchanged or had been modified in light of the responses made by the other members of the panel. This questionnaire was recirculated to each panelist by electronic or postal mail. Subsequent written, e-mail, and personal telephone reminders were made to the nonresponders after 4, 7, 10, and 14 weeks. At each round, a letter explaining the purpose of the exercise accompanied the questionnaire. Two such iterative phases were planned, but if opinion remained disparate and our consensus criteria were not satisfied, an additional round would be conducted. The results were analyzed as part of stage 4 for levels of agreement and degree of consensus.

Statistical advice was taken at each stage of the questionnaire design and was provided by a statistician specializing in educational research at the Medical Education Unit, University of Leeds.

Analysis.

We aimed to establish whether experts in MUS agreed with the issues under consideration and to evaluate levels of agreement between experts with the aim of developing a universally applicable consensus of opinion regarding appropriate practice. Agreement was assessed on 2 levels: 1) Does the expert respondent agree with the issue under consideration; and 2) Does each expert respondent agree with the opinion of other experts on a certain issue (consensus element). A 1–5 Likert scale was used to assess opinion in response to the statements contained in the questionnaire in which 1 indicated “definitely no/definitely not important,” 2 indicated “probably no/probably not important,” 3 indicated “no opinion,” 4 indicated “probably yes/probably important,” and 5 indicated “definitely yes/essential.” Results were expressed as the cumulative percentage of respondents scoring an item either 4 (probably yes/probably important) or 5 (definitely yes/essential) (total cumulative agreement). Group agreement with the issue under consideration was defined as total cumulative agreement ≥70% after the second Delphi round. The net change in agreement between Delphi rounds was used as a measure of the level of agreement between the panel members (net total cumulative agreement). Group consensus was defined as a net change of less than ±10%. If both of these parameters were satisfied, group consensus agreement was established and the category was defined as appropriate. The results have been presented as the total cumulative agreement after each Delphi round and the net total cumulative agreement; they also have been broken down in terms of specialty background. Data evaluation and statistical analysis were carried out by the authors using SPSS version 10 (Chicago, IL) under the direction of a statistician specializing in medical education research. Nonparametric statistical tests were used to assess levels of significance (independent samples were compared using the Mann-Whitney test; related samples were compared using the Wilcoxon's signed rank test).

Composition of experts.

Fifty-seven international experts in MUS were identified who satisfied our selection criteria (Table 1). Eleven were based in the United Kingdom, 28 in mainland Europe (Austria, Belgium, Denmark, Finland, France, Germany, The Netherlands, Italy, Spain, and Switzerland), 13 in North America, and 5 elsewhere (Australia, Asia, South America). This group was composed of 20 rheumatologists and 37 radiologists. Their overall median duration of MUS practice was 8–9 years (interquartile range [IQR] 4–19 years) and individuals performed a median of 4–5 MUS sessions per week (IQR 2–7) on a median of 21–25 patients (IQR 11–35) (17).

Response rate.

The overall response rate from the first Delphi round was 70% (40 of 57). This included a 90% response (18 of 20) from rheumatologists and a 59% response (22 of 37) from radiologists. One reply was anonymous and 1 was incomplete; these were not used. The overall response rate from the second Delphi round was 95% (36 of 38). The relative proportion of these respondents per specialty was 100% (18 of 18) for rheumatologists and 90% (18 of 20) for radiologists. Nonresponders to the first questionnaire were not included in the second round.

Group consensus agreement of appropriate indications for a rheumatologist to perform MUS.

After the second Delphi exercise, the total cumulative agreement scores for monitoring disease activity (89%), monitoring disease progression (87%), bursitis (84%), effusion (84%), inflammatory arthritis (81%), guided aspiration (81%), and guided injection (81%) were all easily above the threshold value of 70%, signifying group agreement (Table 2). The net level of agreement for almost all these indications increased between rounds, implying a positive feeling within the group that these indications were indeed appropriate. The greatest change was +8% for monitoring disease progression, and consequently all categories remained comfortably within our ±10% criteria for consensus agreement. Tendon pathology showed less total cumulative agreement, with a negative net change from 78% to 75% between rounds, but this still fulfilled our criteria to be included as an appropriate indication. Degenerative arthritis (46%), soft tissue mass (46%), nerve lesions (51%), ligament injury (51%), and muscle injury (57%) all scored well below our 70% round-2 limit, signifying insufficient group agreement. The corresponding net total cumulative agreement scores were all negative, implying increasing agreement that these were inappropriate indications for a rheumatologist to perform a MUS examination. Considering the overall change of all scores between rounds, the only indications where there was a statistically significant difference in responses were nerve lesions (P = 0.01), ligament injury (P = 0.02), and monitoring disease progression (P = 0.02). Therefore, appropriate indications for a rheumatologist to perform a MUS examination as defined by expert group consensus are inflammatory arthritis, tendon pathology, joint or soft tissue effusion, bursitis, monitoring disease activity, monitoring disease progression, guided aspiration, and guided injection.

Group consensus agreement of appropriate anatomic areas for a rheumatologist to perform MUS.

Group agreement was established for the hand (81%), wrist (81%), elbow (78%), hip (75%), knee (75%), ankle and heel (75%), forefoot (75%), and shoulder (72%) as appropriate regions for rheumatologists to perform MUS (Table 3). Net agreement was within our defined limits between rounds for all these sites, with the greatest change being +6% in the hand and wrist. Hence, our criteria for group consensus agreement were satisfied in each of these anatomic areas. Soft tissue (56%) and groin (56%) both scored well below our 70% threshold for group agreement, and even though net total cumulative agreement showed little change between rounds, these areas were consequently considered inappropriate for a rheumatologist to perform a MUS examination. The overall change in expert responses between rounds did not reach statistical significance in any anatomic area. Therefore, appropriate anatomic areas for a rheumatologist to perform a MUS examination as defined by expert group consensus are the hand, wrist, elbow, shoulder, hip, knee, ankle and heel, and forefoot.

Group consensus agreement of appropriate knowledge and skills that a rheumatologist requires to perform MUS.

All knowledge and skill categories satisfied our criteria for group agreement, with total cumulative agreement scores ranging between 86% and 100% (Table 4). Net agreement was either positive, with a maximum score of +6% for color Doppler, or did not change between rounds. There were no negative net agreement scores. Consensus agreement was consequently established for all 14 knowledge and skill categories. Statistically significant differences in the overall scoring between rounds were seen in the categories of color Doppler (P = 0.01), machine function and operation (P = 0.046), and image optimization (P = 0.046). Therefore, appropriate knowledge and skills that a rheumatologist requires to perform a MUS examination as defined by expert consensus agreement are ultrasound physics, anatomy, pathology, clinical application and relevance, indications and limitations, artifact, machine function and operation, patient and probe position, planes and system of examination, image optimization, dynamic assessment, color Doppler, and power Doppler (Table 5).

Differences between specialty background.

When we divided responses in terms of specialty background (i.e., rheumatologist or radiologist) some interesting differences were observed. Total cumulative agreement scores for radiologists are consistently lower than those of rheumatologists for all indications and anatomic areas and most knowledge and skill categories, although differences are lowest in this latter section (mean 67% versus 95%; range 16–100% versus 67–100%). Indeed, statistically significant differences were demonstrated between responses given by the 2 specialty groups for all categories of indication and anatomic area after both Delphi rounds except for “monitoring disease activity” in round 1 (P = 0.06). No statistically significant differences were seen between responses for any of the knowledge and skill categories after either round. In the categories where consensus agreement was satisfied, the mean overall difference in total cumulative agreement scores between specialties after round 2 was 22% (range 0–50%), compared with 55% (40–73%) in the categories defined as inappropriate for rheumatologists. The mean cumulative agreement scores for rheumatologists and radiologists were 99% (83–100%) versus 77% (50–100%) for the accepted categories and 79% (67–89%) versus 25% (16–33%) for the rejected categories. Rheumatologists scored 100% total cumulative agreement for 25 of 30 appropriate categories, the exceptions being shoulder (95%), hips (94%), equipment (94%), power Doppler (94%), and physics (83%). However, they scored significantly less for all 7 inappropriate categories: degenerative arthritis (67%), soft tissue mass (72%), groin (78%), soft tissue (78%), ligament injury (83%), nerve lesions (89%), and muscle injury (90%). When reviewing indications and anatomic areas, rheumatologists were more likely to score a category as “definitely important/essential” rather than “probably important”; radiologists on the other hand were much more likely to score a category as “probably important” rather than “definitely important/essential.” Little difference was noted in the categories of knowledge and skills.

If we consider change of opinion between Delphi rounds, the mean net total cumulative agreement was greatest among radiologists (mean +1.5%, range −9 to +11) compared with rheumatologists (mean +0.6%, range −11 to +15). These differences were only statistically significant among the rheumatologists' scores in the anatomic category of nerve lesions (P = 0.046) and in the knowledge and skill category color Doppler (P = 0.03). None of the responses made by the radiologists changed to a statistically significant level between Delphi rounds 1 and 2. In the categories where consensus agreement was satisfied, the mean overall net change in cumulative agreement score was +1.7%, whereas in the categories defined as inappropriate for rheumatologists it was −1.8%. When divided by specialty, the mean net total cumulative agreement score in the accepted categories among rheumatologists was +2.2% compared with +1.6% for radiologists. In the rejected categories, the mean net change was −6.2% among rheumatologists and +1.0% for radiologists.