Communicating accuracy of tests to general practitioners: a controlled study
BMJ 2002; 324 doi: https://doi.org/10.1136/bmj.324.7341.824 (Published 06 April 2002) Cite this as: BMJ 2002;324:824
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
I read this article with interest. I am currently involved in a pilot RCT of a management algorithm for stable angina, along with my colleagues, Dr. David White, Dr. Neil Drummond, and Jason Weshler. We have programmed the ACC/AHA clinical practice guideline <_1/> for the Palm Handheld computer, and are evaluating this in primary care practices.
According to Bayes'theorem, a test's usefulness is maximal when the pre-test probability of a condition is intermediate. Steurer et al found that family physicians had difficulty estimating the pre-test probability using clinical criteria (such as age), and over-estimated post-test probability in a low prevalence condition. We present the pre-test probability of angina to the clinician at the bedside, using palm-based software, and the program suggests further investigations using a Bayesian approach.
Perhaps some of our difficulties with EBM can be overcome by having easy to use software at the bedside to help us with probability calculations and evidence-based management. I hope that this trial will give us some indication as to whether this approach makes sense.
Michelle Greiver, MD, CCFP
1. Gibbons RJ, Chatterjee K, Daley J, Douglas JS, Fihn SD, Gardin JM, Grunwald MA, Levy D, Lytle BW, O'Rourke RA, Schafer WP, Williams SV. ACC/AHA/ACP-ASIM guidelines for the management of patients with chronic stable angina: executive summary and recommendations: a report of the American College of Cardiology/American Heart Association Task Force on practice guidelines (Committee on management of patients with chronic stable angina). Circulation 1999;99:2829-48
Competing interests: No competing interests
Sir,
I congratulate Steurer et al (ref. 1) for their excellent study showing encouraging results when they presented the positive likelihood ratio in a non-technical language. I have observed while conducting evidence based medicine workshops that the health professionals find the concept of likelihood ratio difficult unless presented in a plain language. They also find it difficult to follow the three steps to determine posttest probability using the pretest probability and likelihood ratio.
Most authors (refs. 2, 3) including Steurer et al (ref 1)first convert pretest probability into pretest odds (step 1), then determine posttest odds by multiplying the pretest odds with the likelihood ratio (step 2) and then convert the posttest odds to the posttest probability (step3). This method requires explaining the concept of odds and its relation with probability, which the health professionals find difficult to follow. As a solution to this, I have derived the following formula, which gives posttest probability (Post-TP)from the pretest probability (Pre-TP) and likelihood ratio (LR)in one step:
Post-TP = (Pre-TP x LR)/[1 + Pre-TP (LR - 1)]
The formula is derived from the formula used in step 2 of the three- step method using simple algebra. This formula does not require use (hence explaining)of the term odds. I understand that Steurer et al(ref 1) did not go for explaining this calculation to the general practitioners. But they indicated that they used the three-step method to do their calculation. It is in this context that I present the above formula. I have found it useful in communicating the calculation to the health professionals. I recommend its use in future write-ups and workshops on evidence-based medicine.
References: 1. Steurer J, Fischer JE, Bachmann LM, Koller M, Rier G. Communicating accuracy of tests to general practitioners: a controlled study. 2. Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine. 2nd ed. Boston, Mass: Little, Brown & Co; 1991. 3. Jaeschke R, Guyatt GH, Sackett DL for the Evidence-Based Medicine Working Group. Users'guides to the medical literature, III. how to use an article about a diagnostic test, B: What are the results and will they help me in caring for my patients? JAMA 1994;271:703-707.
Competing interests: No competing interests
The study by Steurer and colleagues provides further evidence of the difficulty that clinicians experience in applying information about the sensitivity and specificity of a diagnostic test. I support their call for a different way of expressing the reliability of a test but I disagree with their conclusion that, in their study, “most general practitioners recognised the correct definitions for sensitivity and positive predictive value but did not apply them correctly”. In the vignette, the GPs were not given the positive predictive value (PPV), and so did not have a chance to apply it. Had they done so, I suspect that they would have performed better than they did when given the positive likelihood ratio.
The group of GPs who were given the information that the positive likelihood ratio was two did better than the other groups, but they still did not do that well. Their estimates of the chance of cancer ranged from an attributed likelihood ratio of less than 2 to one of almost 30. Had they been given the PPV of the test for the diagnosis of carcinoma in a woman of 64 with abnormal uterine bleeding, it is hard to see that many would have failed to give the correct answer to the question: “what is the probability that this woman has endometrial carcinoma” since, by definition, that is the PPV.
The problem with the PPV for any one test, of course, is that it will vary according to the prevalence, which is in turn dependant, in this case, on the age of the patient, her symptoms, and the setting in which she is seen. That can be coped with by giving two or more PPVs, each one related to women with abnormal uterine bleeding seen in primary care at different ages, say 25 and 65. From these the GP can extrapolate to apply the PPV to the patient in question. GPs consult without having access to information about pre-test probabilities and usually without a nomogram from which to calculate post-test probabilities, if given the likelihood ratio. A small range of PPVs for each test would be of more use to them in the real world of primary care.
Competing interests: No competing interests
Understanding Probability Information
We congratulate Steurer and colleagues(1) on an excellent and timely study. It has long been established in the cognitive psychology literature that people have difficulties understanding probability information(2). More recently, research has taken a more positive angle, looking at how probabilities can be communicated more effectively and how people can be taught to interpret them(3)(4). Unfortunately these findings have rarely been applied to actual health screening as done by Steurer et al.
Our recent study of medical undergraduates demonstrated the difficulties of integrating information on test sensitivity and specificity, and the base rates of conditions. 194 students were presented with one of four scenarios giving the same numerical information in slightly different ways. The situation described was either health- related (prenatal serum screening for Down’s syndrome) or neutral (screening engine parts for faults), and the information was either presented as percentages (e.g. 90%) or frequencies (e.g. 9 out of 10). Participants were asked to estimate the probability that a positive test result indicated the presence of an abnormality, given the sensitivity of the test (90%), the false positive rate (i.e. 1-specificity) (1%), and the base rate of abnormalities (1%) (adapted from(2)). (Participants were informed that numbers were for the purpose of the study only).
It was hypothesised that presenting the information as a health scenario in percentages would result in more pronounced and consistent errors. Only 5.2% of participants made correct estimates (10 out of 194), too few to allow valid chi-square analysis of differences between scenarios. The incorrect responses clustered around 1% and 90%, apparently a consequence of participants paying either too little or too much attention to the base rate (i.e. 1%, the prior probability of any individual having the abnormality). Base rate neglect was the most common error when information was presented as percentages, and over-attention to the base rate was most common in the frequency conditions. A smaller study of midwifery students showed similar patterns of errors. These tendencies towards certain answers suggest that the information is not interpreted in the same way if it is presented as frequencies rather than percentages, however neither presentation makes the reader more likely to interpret it correctly.
Work continues on the responses of healthcare professionals, and pregnant women attending a booking-in appointment. If informed uptake of screening tests is to be achieved, then methods of increasing the understanding of professionals and users must be sought.
Helen Adams, Research Student Dr Ros Bramwell, Senior Lecturer, Department of Clinical Psychology, University of Liverpool
Corresponding author: Helen Adams, Department of Clinical Psychology, Whelan Building, Quadrangle, Brownlow Hill, Liverpool. L69 3GB UK tel. 0151 794 4160 fax. 0151 794 5537 email: helena1@liv.ac.uk
References 1. Steurer J, Fischer J, Bachmann L, Koller M, ter Riet G. Communicating accuracy of tests to general practitioners: a controlled study. BMJ 2002,324:824-6
2. Hammerton MA. Case of Radical Probability Estimation. J Exp Psychol 1973;101(2):252-4
3. Gigerenzer G, Hoffrage U. How to Improve Bayesian Reasoning Without Instruction: Frequency Formats. Psychol Rev 1995;102(4):684-704
4. Sedlmeier P, Gigerenzer G. Teaching Bayesian Reasoning in Less Than Two Hours. J Exp Psychol 2001;130(3):380-400
Competing interests: No competing interests