Impact analysis studies of clinical prediction rules relevant to primary care: a systematic review
==================================================================================================

* Emma Wallace
* Maike J M Uijen
* Barbara Clyne
* Atieh Zarabzadeh
* Claire Keogh
* Rose Galvin
* Susan M Smith
* Tom Fahey

## Abstract

**Objectives** Following appropriate validation, clinical prediction rules (CPRs) should undergo impact analysis to evaluate their effect on patient care. The aim of this systematic review is to narratively review and critically appraise CPR impact analysis studies relevant to primary care.

**Setting** Primary care.

**Participants** Adults and children.

**Intervention** Studies that implemented the CPR compared to usual care were included.

**Study design** Randomised controlled trial (RCT), controlled before–after, and interrupted time series.

**Primary outcome** Physician behaviour and/or patient outcomes.

**Results** A total of 18 studies, incorporating 14 unique CPRs, were included. The main study design was RCT (n=13). Overall, 10 studies reported an improvement in primary outcome with CPR implementation. Of 6 musculoskeletal studies, 5 were effective in altering targeted physician behaviour in ordering imaging for patients presenting with ankle, knee and neck musculoskeletal injuries. Of 6 cardiovascular studies, 4 implemented cardiovascular risk scores, and 3 reported no impact on physician behaviour outcomes, such as prescribing and referral, or patient outcomes, such as reduction in serum lipid levels. 2 studies examined CPRs in decision-making for patients presenting with chest pain and reduced inappropriate admissions. Of 5 respiratory studies, 2 were effective in reducing antibiotic prescribing for sore throat following CPR implementation. Overall, study methodological quality was often unclear due to incomplete reporting.

**Conclusions** Despite increasing interest in developing and validating CPRs relevant to primary care, relatively few have gone through impact analysis. To date, research has focused on a small number of CPRs across few clinical domains only.

*   risk prediction
*   clinical prediction rule
*   impact analysis

### Strengths and limitations of this study

*   Clinical prediction rules (CPRs) are increasingly developed and advocated for use in clinical practice. However, little is known regarding the effectiveness of these tools versus usual care for relevant clinical outcomes.

*   This is the first systematic review of CPRs relevant to primary care that have gone through impact analysis.

*   The main limitation of this review is that the electronic search was limited to 30 prespecified journals, which may mean that some relevant studies were not retrieved. This search was supplemented with key author searches and reviewing other resources known to publish CPRs.

*   Nevertheless, this is the first study to examine, in detail, impact analysis studies of CPRs relevant to primary care.

## Introduction

Clinical prediction rules (CPRs) are clinical tools that quantify the individual contributions that various components of the history, physical examination and investigations contribute towards diagnosis, prognosis or likely response to treatment in a patient.1 These tools simplify, standardise and attempt to increase the accuracy and consistency of clinicians’ diagnostic and prognostic assessments and management decisions.1 ,2 Well-recognised examples of CPRs include the Framingham cardiovascular risk score, the Ottawa ankle rule and the Centor score for sore throat.

Developing and validating a CPR requires reference to specific methodological standards.1 ,3 ,4 Conventionally, these tools go through three distinct stages prior to full implementation in a clinical setting.1 ,3 ,4 The first stage is derivation, where the independent and combined effects of explanatory variables such as symptoms, signs and/or investigations, are established. The next stage is validation, where the final derived CPR is evaluated first in a similar clinical setting (internal validation), followed by different clinical settings (external validation). If in following these stages predictive accuracy is established, then the final stage of evaluation is to test the impact of using the CPR in clinical practice, ideally in a randomised controlled trial (RCT) for relevant clinical outcomes.1 ,5 ,6 Impact analysis aims to investigate if the implementation of a CPR in clinical practice is better than usual care for the patient, process of care and/or cost outcomes.7 ,8

Our research group recently published two studies detailing the development and content of an international register of CPRs relevant to primary care.2 ,9 With increasing interest in CPRs, large numbers have been derived but fewer have been validated or tested in an impact analysis study.2 If CPRs are to truly improve the quality of patient care then evaluation of these tools is crucial.

The aim of this systematic review is to present a narrative and critical analysis of CPRs relevant to primary care which have gone through impact analysis.

## Methods

The methods used for identifying CPRs from the literature and in developing a register of these tools relevant to primary care have been published in detail previously.2 ,9 These methods are summarised below.

### Search strategy

An electronic search string for PubMed was developed to retrieve CPRs relevant to primary care from 30 preselected medical journals (see online supplementary appendix 1 for search string and journals included).9 No restriction was placed on language. Original electronic searches were conducted from 1980 to 2009, which for the purposes of this review were updated to the end of 2013.2 In addition, secondary sources of CPRs were searched including the *Journal of the American Medical Association* (*JAMA*) Rational Clinical Examination series, a handbook of CPRs and personal resources.2 ,9 ,10 Author searches for key experts in the field were also conducted for additional relevant articles. Furthermore, reference lists of each relevant impact analysis study were searched to identify possible additional studies.

### Supplementary appendix

[[bmjopen-2015-009957supp.pdf]](pending:yes)

### Inclusion criteria

Studies were eligible for inclusion if they met the following criteria.

(1) *Population*: Relevant to primary care defined as ‘normally the point of first medical contact within the healthcare system, providing open and unlimited access to its users, dealing with all health problems regardless of the age, sex or any characteristic of the person concerned’.11 Although studies may not have been conducted in a primary care setting, they were eligible for inclusion providing they were relevant to primary care (eg, implementation of the Canadian head CT rule with the aim of reducing imaging for patient presenting with minor head injury). This inclusion criterion was designed to be broad to acknowledge variation in the same-day diagnostic tests that are available across different countries and the international variation in the role of primary care clinicians. Studies set in the emergency department were considered relevant to primary care if following application of the CPR the patient could be discharged home following application of the CPR.

(2) *Intervention*: CPR defined as ‘a clinical tool that quantifies the individual contributions that various components of the history, physical examination and investigations make towards the diagnosis, prognosis or likely response to treatment in a patient’.1 Diagnostic, prognostic and management CPRs were included and screening questionnaires (ie, applied to apparently healthy people who may be at increased risk of a disease or condition) were excluded. A requirement for inclusion was that the CPR comprised the entire intervention. Studies where the CPR was implemented as part of a broader guideline, protocol or decision aid were excluded. Studies that used a CPR to determine eligibility for trial inclusion but were not part of the intervention were also excluded.

The following study designs were included: (cluster) RCT, controlled before–after or interrupted time series studies. Uncontrolled study designs were excluded as the aim of this review was to examine the effectiveness of CPR implementation, rather than the performance of the CPR which would be captured in validation studies using observational study designs.

(3) *Comparison*: Usual care.

(4) *Primary Outcome*: Physician behaviour, for example, ordering of diagnostic tests; process of care, for example, number of inpatient bed days; and/or patient outcomes, for example, duration of symptoms.1

### Data extraction

All articles were initially screened for inclusion according to title and abstract by one reviewer. Potentially relevant articles were then reviewed by a second reviewer with any disagreements resolved by a third independent reviewer. For each relevant article, the following data was extracted: (1) name of CPR; (2) type of CPR: prediction rule, decision rule or both; (3) clinical domain: using International Classification of Primary Care—second edition (ICPC-2)12; (4) clinical setting; (5) study population; (6) primary outcome of interest; (7) predictive accuracy of the CPR (measured by sensitivity/specificity (95% CIs) where reported, otherwise, the model's c-statistic was recorded) and (8) impact on primary outcome of interest.

### Data analysis

#### Critical analysis of CPR impact analysis

Each article was critically appraised, using a published framework for impact analysis of CPRs.7 Developed in 2011 by an expert panel, this four-phase framework provides guidance for impact analysis studies (see figure 1). The phases are as follows: (1) *Exploratory phase*; evaluate the level of evidence and predictive accuracy of the CPR; (2) *Preparation for impact analysis*; consider potential barriers, assess acceptability of the CPR to clinicians and local stakeholders and conduct a pilot study; (3) *Experimental phase*; evaluation of the CPR with monitoring of the use of the CPR in a clinical setting; (4) *Long-term implementation phase*; examine if a CPR with reported positive impact on relevant clinical outcomes is implemented long-term and how this was achieved.7

![Figure 1](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/6/3/e009957/F1.medium.gif)

[Figure 1](http://bmjopen.bmj.com/content/6/3/e009957/F1)

Figure 1 
Framework for the impact analysis and implementation of clinical prediction rules (CPRs).7

#### Summary of effect on process and outcome of care

Meta-analysis was not possible due to heterogeneity in CPRs and outcomes of interest, so a narrative analysis was conducted. In this section, wherever appropriate and where data was available, crude ORs and absolute risk reductions (ARR) were calculated.

### Methodological quality assessment

The methodological quality of each impact analysis study was independently evaluated by two reviewers (MU and BC), and by a third reviewer if consensus was not reached (EW). For each study design, an appropriate quality assessment checklist was used. RCTs and cluster RCTs were assessed through the Cochrane risk of bias tool.13 Controlled before–after studies, and interrupted time series studies, were evaluated through Cochrane criteria for these study designs.14

## Results

### Overview of studies

#### Study identification

A flow diagram of the search strategy is presented in figure 2. The PubMed search (1980–2013) and supplementary sources searches retrieved a total of 86 158 studies, of which 1111 CPR studies were identified following review of title and abstract. A total of 18 studies met the inclusion criteria for the systematic review.

![Figure 2](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/6/3/e009957/F2.medium.gif)

[Figure 2](http://bmjopen.bmj.com/content/6/3/e009957/F2)

Figure 2 
Flow diagram of search strategy.

#### Description of included impact analysis studies

A total of 14 unique CPRs were tested in 18 impact analysis studies (table 1 and online supplementary appendix 2). According to ICPC-2, these studies were classified into four broad clinical domains, namely, musculoskeletal, most commonly the Ottawa ankle rule15–20 (n=6); cardiovascular21–27 (n=6); respiratory28–32 (n=5); and neurological33 (n=1). The majority of studies were conducted in North America (Canada n=10, the USA n=4), with the remainder in the UK (n=2) and France (n=2). Most studies were set in the emergency department (ER) (n=9) and primary care (n=7). The remainder were carried out in the outpatient department (n=2).

View this table:
[Table 1](http://bmjopen.bmj.com/content/6/3/e009957/T1)

Table 1 
Summary of impact analysis studies of CPRs relevant to primary care

Regarding study design, there were four cluster RCTs,15 ,18 ,27 ,33 eight RCTs,22 ,24–26 ,28–32 one pilot RCT,23 three controlled before–after studies,16 ,17 ,19 and two interrupted time series.20 ,21 In a total of 16 studies, the intervention was the impact of the CPR alone,15–29 ,32 ,33 and two studies used different trial arms to test the CPR alone versus CPR and protocol versus usual care.30 ,31 Two studies integrated the CPR into a computerised clinical decision support system (CDSS).20 ,32 Two studies used real-time CPR reminders at the point of test ordering.18 ,33

##### Critical analysis of CPR impact analysis studies

###### Preparation for impact analysis: level of evidence of CPR, consideration of potential barriers and assessment of CPR acceptability

Fifteen of 18 studies implemented a CPR that was externally validated, while three studies tested a CPR that had been derived or internally validated only.21 ,30 ,31 Ten studies reported the CPR's sensitivity from validation studies in identifying the target outcome which ranged from 85% to 100%.15–20 ,28 ,29 ,31 ,33 Five studies identified and addressed potential barriers for implementation before impact analysis, most frequently through barriers analysis.18 ,20 ,22 ,32 ,33 Six studies assessed the acceptability of the CPR to clinicians prior to the implementation phase of the study, usually through training sessions and engagement with local stakeholders.15 ,19 ,22 ,27 ,32 ,33 Seven studies reported that a pilot or simulation phase was conducted or there was a previous impact analysis on the same CPR by the same authors.16–19 ,21 ,32 ,33

###### Impact analysis phase: adherence with CPR use and reasons for non-adherence

Twelve studies tracked the use of the CPR during implementation, usually with standardised data collection forms or computerised tools.15–18 ,20 ,22 ,25 ,26 ,29 ,31–34 Overall, adherence with CPR use varied between studies ranging from 57.5% to 100%, with reported reasons for non-adherence including fear of missing the diagnosis, preference for own clinical judgement and patient request.20 ,33 Of the 12 studies that tracked CPR adherence, seven reported adherence of ≥80%, four reported adherence of 60–80%, and one study reported adherence of ≤60%. Nine of 12 studies reported a positive impact on primary study outcome following CPR intervention, but there was no clear link between level of adherence and successful CPR implementation. Clinicians’ acceptability of CPR use during the intervention phase was evaluated in four studies,16 ,17 ,20 ,33 of which two assessed the reported rate of comfort using a five-point Likert scale.16 ,17

###### Postimplementation phase: maintaining use of CPR

Of 10 studies with a positive impact on primary outcome, 4 evaluated the effect of the CPR in a post-intervention phase ranging from 5 to 12 months.15 ,16 ,18 ,20 To maintain CPR use, two studies used a passive strategy of posters, one retained computerised clinical decision support, and one did not employ any particular strategy. In all four studies, the effect of CPR use was maintained.

#### Importance of study design in assessing the impact of CPRs

There were five uncontrolled before–after studies retrieved during the initial search which were excluded from data analysis based on their uncontrolled study design (see online supplementary appendix 3).35–39 These studies tested the impact of the Ottawa ankle rule (n=2), the Canadian C-spine rule (n=1), the CT head rule (n=1), and the Glasgow Blatchford bleeding score (n=1). All five studies demonstrated a positive impact on primary outcome, usually physician behaviour in ordering imaging.

##### Effect on the process and outcome of care

Overall, 10 studies reported that CPR implementation resulted in a positive impact on primary outcome, while eight studies reported no impact versus usual care. There were no clinically important adverse outcomes reported. Studies are presented according to clinical domain. Table 2 presents a summary of the estimated effect sizes for the impact analysis studies.

View this table:
[Table 2](http://bmjopen.bmj.com/content/6/3/e009957/T2)

Table 2 
Table of estimated effect sizes for impact analysis studies

#### Musculoskeletal (cluster RCTs n=2, controlled before–after studies n=3, interrupted time series n=1)

All six musculoskeletal studies focused on the implementation of CPRs for deciding on further imaging for patients presenting with ankle, knee or neck injury.15–20 All the included musculoskeletal CPRs had reported sensitivities of 100% in their validation studies, and they all focused on physician behaviour in deciding to order imaging. Of these six studies, five reported a positive effect on reducing imaging with crude ORs ranging from 0.03 to 0.9615–18 ,20 (see table 2). All studies adopted an educational approach to encourage CPR use among clinicians through use of educational meetings, posters and pocket cards. Of note, three studies tested the impact of the Ottawa ankle rule; two controlled before–after trials in Canada, and one cluster RCT in France.15 ,16 ,19

#### Cardiovascular (cluster RCT n=1, RCTs n=4, interrupted time series n=1)

Of six cardiovascular studies, two implemented chest pain CPRs to assess the impact on physician decision-making regarding emergency admission for patients with suspected myocardial infarction.21 ,22 One of these studies reported a 30% relative reduction in patients admitted inappropriately.21 The remaining four studies implemented cardiovascular risk scores in general practice. Three of these studies reported no impact on physician behaviour such as prescribing and referral to dieticians, or on patient outcomes such as reduction in lipid levels.23 ,24 ,27 However, in one large-scale RCT (n=3053) that published its findings in two separate articles, both patient lipid levels and physician antihypertensive prescribing were improved.25 ,26

#### Respiratory (RCTs n=5)

Of five respiratory studies, four focused on physician behaviour in terms of antibiotic prescribing for sore throat in general practice.28–30 ,32 Of these four studies, only one reported significantly reduced antibiotic prescription rates in the intervention group (age-adjusted relative risk 0.74, 95% CIs 0.60 to 0.92) versus usual care.32 The primary outcome in the fifth study was reported symptom severity in patients presenting with sore throat, and antibiotic prescribing was included as a secondary outcome.31 This study found that use of the CPR alone or CPR in combination with a rapid antigen detection test improved patient-reported symptom severity and duration, and reduced antibiotic use by 29% (adjusted risk ratio 0.71, 95% CI 0.50 to 0.95).31

#### Neurological (cluster RCT n=1)

One study implemented the Canadian CT head rule, which guides the ordering of brain imaging in patients presenting following minor head injury.33 Despite this CPR having 100% sensitivity in validation studies, it did not reduce imaging rates. In process evaluation, clinicians’ reported unease with certain components of the rule and fear of missing a high-stakes diagnosis as reasons for not adopting the CPR.33

### Methodological quality assessment of included studies

Studies were heterogeneous with regard to risk of bias. For the RCT designs (n=13), five studies were considered low risk of bias for random sequence generation, and five were considered low risk in relation to allocation concealment (figure 3A). The remaining RCT studies had an unclear risk in these domains. Owing to the nature of many of the interventions, it was not always possible to blind participants and research personnel, therefore, performance bias was judged to be unclear or high in over half these studies. In the non-randomised study designs, the risk of selection bias was high in all studies while the risk of blinding and contamination was low in all studies (see figure 3B). Overall, six studies tested the impact of a CPR in which the authors were involved in developing.17 ,18 ,22 ,29 ,31 ,33 The impact that this may have in terms of bias is unclear.

![Figure 3](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/6/3/e009957/F3.medium.gif)

[Figure 3](http://bmjopen.bmj.com/content/6/3/e009957/F3)

Figure 3 
(A): Methodological quality assessment of impact analysis studies with RCT study design. (B) Methodological quality assessment of impact analysis studies with controlled before-after study design.

## Discussion

### Summary of main findings

This review indicates that despite the increasing research interest in developing and validating CPRs, relatively few of these tools relevant to primary care have gone through impact analysis. Implementation has been restricted to a few clinical domains mainly musculoskeletal, cardiovascular and respiratory, and certain CPRs have undergone multiple evaluations, for example, the Ottawa ankle rule. Of 18 studies meeting inclusion criteria, 10 demonstrated an improvement in primary outcome with CPR use when compared to usual care. Approximately half these successful studies focused on changing physician behaviour in ordering imaging for patients presenting with ankle, knee and neck musculoskeletal injuries.15–18 ,20 Four studies with a positive impact on the study's primary outcome successfully implemented post-RCT measures to maintain the impact through both passive (posters) and active strategies (retention of computerised CDSS).15 ,16 ,18 ,20

Studies which aimed to alter physician behaviour regarding prescribing were less successful with three of six such studies successful in reducing prescription rates.23 ,31 ,32 Studies that reduced antibiotic prescription rates invested significant time before CPR implementation in assessing acceptability to clinicians, and also integrated the CPR into the clinical work flow through computerised clinical decision support or point-of-care reminders.31 ,32 The importance of this type of impact analysis preparation in adequately addressing barriers to implementation, and in integrating the CPR into the clinical workflow has been highlighted.5 ,7 ,40 In this review, 12 studies considered barriers to implementation and/or gauged the acceptability of the CPR to clinicians prior to impact analysis. However, only four studies integrated the CPR into clinical work flow using either computerised CDSS or point-of-care reminders.18 ,20 ,32 ,33

The perceived seriousness of the target condition may also affect implementation of CPR. For instance, the impact of the Canadian CT head rule was evaluated in the diagnostic pathway of intracranial bleeding following minor head injury.33 This CPR has 100% sensitivity, and though implemented by an experienced CPR research group, this CPR did not impact on CT imaging rates.33 In a parallel process evaluation, clinicians’ reported unease with certain components of the rule, and fear of missing a high-stakes diagnosis as reasons for not adopting the CPR.33

Overall, adherence with CPR use during implementation varied considerably between studies ranging from 57% to 100%. Reasons for non-adherence, established through process evaluation, related to fear of missing the diagnosis, preference for own clinical judgement, and patient request for further investigation or management.

### Comparison with existing literature

Previous CPR reviews relevant to inpatient and paediatric settings reported issues around the variability of methodological quality in conducting CPR studies and a paucity of impact analysis studies.3 ,6 ,41 The issue of methodological quality has recently been addressed with the publication of two standardised reporting guidelines for CPR derivation and validation studies, and systematic reviews of CPRs.42 ,43 These publications will have an important role to play in standardising CPR research and in promoting robust validation of CPRs which should then be prioritised for evaluation in future impact analysis studies.

CPRs, which demonstrate improvements in the process of care and/or patient outcomes, should then be considered for inclusion in relevant clinical guidelines to facilitate dissemination into clinical practice. A recent survey which examined the use of CPRs in clinical practice by GPs in the UK reported that GPs most often used cardiovascular, depression, fracture and atrial fibrillation CPRs.44 CPR use was dictated by perceived clinical utility, familiarity and local policy requirements. In a supplementary review of clinical guidelines, very little interguideline consistency was found to guide clinicians in terms of which, if any, of these tools to use in practice.44 Prioritising the evaluation of a few adequately validated CPRs with proven predictive accuracy in relevant clinical settings would add significantly to this evidence base and facilitate, if appropriate, the inclusion of certain CPRs into future clinical guidelines.

### Implications for clinical practice and research

CPR research is a relatively new methodological discipline and a challenging area of research.2 In the conduct of this review, several uncontrolled before–after impact analysis studies were retrieved. While these studies have a role in contributing to the overall evidence base, they are not a substitute for carefully conducted RCTs in determining the effectiveness of CPRs on clinically relevant outcomes.6 In this review, the majority of included RCTs focused on physician behaviour or process of care as the primary outcome. This is not surprising considering how challenging it is to demonstrate differences in patient outcomes, requiring much larger sample sizes which significantly increase running costs.10 In addition, contextual issues which exist between countries, due to differences in healthcare delivery, healthcare systems and incentives, render process of care outcomes difficult to generalise.5 ,6

Certainly, CPR impact needs to be considered early in the development phase of any new CPR. For instance, Irish research shows high levels of GP referrals to symptomatic breast units.45 Recent research efforts have focused on the development of a breast cancer CPR for use in primary care to aid these referral decisions.46 However, although this CPR underwent methodologically robust development and demonstrates good predictive accuracy it is unlikely its use will impact on referral rates. This is due largely to the existence of a low-risk threshold for referral driven by a combination of factors including patient expectation, media interest and fears of medicolegal ramifications for clinicians if a diagnosis is missed. So when considering an impact analysis RCT in this clinical domain, these contextual issues would need to be addressed in tandem with validation and impact analysis studies.

Certain clinical domains have seen a proliferation of CPR research, particularly musculoskeletal and cardiovascular conditions. The publication of several carefully conducted impact analysis trials for CPRs relating to knee, ankle and neck injuries is largely due to one Canadian research group, while historically, the availability of large UK population data sets facilitated the development of cardiovascular prognostic CPRs.15–18 ,47 ,48 In this review, five impact analysis studies (two were uncontrolled before–after studies detailed in online supplementary appendix 3) focused on the impact of the Ottawa ankle rule in emergency room settings, three of which were conducted in the same country.15 ,16 ,19 ,35 ,36 Ideally, CPR development and impact analysis should be aligned with clinical need rather than developing or testing the effectiveness of CPRs when accurate tools already exist.44

The relatively small number of impact analysis studies retrieved means it is not possible to make firm conclusions about the overall effectiveness of these tools in primary care. However, certain CPRs, such as the Ottawa ankle and knee rules, are appropriate for use in clinical practice, and have a role in reducing unnecessary imaging rates. Future research should focus on conducting RCTs of broadly validated CPRs with consideration of contextual and local implementation factors.7 Pertinent issues include how best to integrate the CPR into clinical workflow, and the potential benefits of embedding CPRs as part of computerised clinical decision support.

### Study limitations

Although this review was conducted systematically, and multiple resources searched to retrieve relevant articles, electronic searches were limited to 30 preselected journals, and as a result it is possible that relevant studies were not retrieved. In addition, this search was last updated in December 2013. However, to the best of our knowledge, this review is the first to analyse, in detail, CPR impact analysis studies relevant to primary care. The broad definition of primary care used for this review led to the inclusion of impact analysis studies conducted in the emergency room setting. This was necessary to account for the variation in primary care services and access internationally. Studies that implemented CPRs as part of a broader guideline, protocol or decision aid were excluded. Finally, due to the heterogeneous nature of the included studies, meta-analysis was not possible.

## Conclusion

Impact analysis of CPRs in primary care has, to date, focused on a small number of CPRs in a limited number of clinical domains. Future research should focus on prioritising well-validated and accurate CPRs for impact analysis to determine if these tools impact on the process of clinical care and patient outcomes.

## Footnotes

*   EM and MJMU are co-first authors and contributed equally.

*   Contributors EW wrote the manuscript, acted as a reviewer during the systematic review and contributed to the results. MJMU contributed to writing the manuscript, acted as a reviewer for the systematic review, contributed to the results, and did methodological quality assessment. BC acted as a reviewer for the systematic review, contributed to the results and completed the methodological quality assessment of included articles. AZ acted as a reviewer for the systematic review and contributed to the results. CK ran the original electronic searches, acted as a reviewer, and contributed to the results and discussion. RG acted as a reviewer, and contributed to the results and discussion. SMS acted as a reviewer, and contributed to the results and discussion. TF conceived the idea, oversaw the project and acted as a reviewer. All authors read and approved the final manuscript.

*   Funding This research was funded by the Health Research Board (HRB) of Ireland through the HRB Centre for Primary Care Research, grant number HRC/2007/1.

*   Competing interests None declared.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data sharing statement No additional data are available.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)

## References

1.  McGinn TG, Guyatt GH, Wyer PC, et al. Users’ guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 2000;284:79–84. [doi:10.1001/jama.284.1.79](http://dx.doi.org/10.1001/jama.284.1.79)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.284.1.79&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=10872017&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000087843000029&link_type=ISI) 

2.  Keogh C, Wallace E, O'Brien KK, et al. Developing an international register of clinical prediction rules for use in primary care: a descriptive analysis. Ann Fam Med 2014;12:359–66. [doi:10.1370/afm.1640](http://dx.doi.org/10.1370/afm.1640)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiYW5uYWxzZm0iO3M6NToicmVzaWQiO3M6ODoiMTIvNC8zNTkiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi8zL2UwMDk5NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  Wasson JH, Sox HC, Neff RK, et al. Clinical prediction rules. Applications and methodological standards. N Engl J Med 1985;313:793–9. [doi:10.1056/NEJM198509263131306](http://dx.doi.org/10.1056/NEJM198509263131306)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1056/NEJM198509263131306&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=3897864&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1985ARA8900006&link_type=ISI) 

4.  Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 1997;277:488–94. [doi:10.1001/jama.1997.03540300056034](http://dx.doi.org/10.1001/jama.1997.03540300056034)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.1997.03540300056034&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=9020274&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1997WG05400037&link_type=ISI) 

5.  Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 2006;144:201–9. [doi:10.7326/0003-4819-144-3-200602070-00009](http://dx.doi.org/10.7326/0003-4819-144-3-200602070-00009)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-144-3-200602070-00009&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=16461965&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000235074800007&link_type=ISI) 

6.  Toll DB, Janssen KJ, Vergouwe Y, et al. Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 2008;61:1085–94. [doi:10.1016/j.jclinepi.2008.04.008](http://dx.doi.org/10.1016/j.jclinepi.2008.04.008)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2008.04.008&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=19208371&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000259955800003&link_type=ISI) 

7.  Wallace E, Smith SM, Perera-Salazar R, et al. Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs). BMC Med Inform Decis Mak 2011;11:62. [doi:10.1186/1472-6947-11-62](http://dx.doi.org/10.1186/1472-6947-11-62)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1186/1472-6947-11-62&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=21999201&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 

8.  Moons KG, Altman DG, Vergouwe Y, et al. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 2009;338:b606. [doi:10.1136/bmj.b606](http://dx.doi.org/10.1136/bmj.b606)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzMzgvanVuMDRfMi9iNjA2IjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpvcGVuLzYvMy9lMDA5OTU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

9.  Keogh C, Wallace E, O'Brien KK, et al. Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish a Web-based register. J Clin Epidemiol 2011;64:848–60. [doi:10.1016/j.jclinepi.2010.11.011](http://dx.doi.org/10.1016/j.jclinepi.2010.11.011)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.11.011&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=21411285&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 

10. Ebell MH. Evidence-based diagnosis: a handbook of clinical prediction rules. New York: Springer-Verlag, 2001.
    
    

11. Allen J, Gay BG, Crebolder H, et al. The European Definition of General Practice/Family Medicine. Short version. WONCA Europe, 2011. [http://www.unige.ch/medecine/umpr/Enseignement/stage3ememaster/Definition\_wonca\_tree.pdf](http://www.unige.ch/medecine/umpr/Enseignement/stage3ememaster/Definition_wonca_tree.pdf) (accessed 18 Mar 2014).
    
    

12. WONCA International Classification Committee. The international classification of primary care. 2nd edn. [http://www.kith.no/upload/2705/icpc-2-english.pdf](http://www.kith.no/upload/2705/icpc-2-english.pdf) (accessed: 22 Mar 2014).
    
    

13. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. [doi:10.1136/bmj.d5928](http://dx.doi.org/10.1136/bmj.d5928)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDMvb2N0MThfMi9kNTkyOCI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi82LzMvZTAwOTk1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

14. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. March, 2011. [http://handbook.cochrane.org/](http://handbook.cochrane.org/)
    
    

15. Auleley GR, Ravaud P, Giraudeau B, et al. Implementation of the Ottawa ankle rules in France. A multicenter randomized controlled trial. JAMA 1997;277:1935–9. [doi:10.1001/jama.1997.03540480035035](http://dx.doi.org/10.1001/jama.1997.03540480035035)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.1997.03540480035035&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=9200633&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1997XF08700037&link_type=ISI) 

16. Stiell IG, McKnight RD, Greenberg GH, et al. Implementation of the Ottawa ankle rules. JAMA 1994;271:827–32. [doi:10.1001/jama.1994.03510350037034](http://dx.doi.org/10.1001/jama.1994.03510350037034)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.1994.03510350037034&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=8114236&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1994MZ49800028&link_type=ISI) 

17. Stiell IG, Wells GA, Hoag RH, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA 1997;278:2075–9. [doi:10.1001/jama.1997.03550230051036](http://dx.doi.org/10.1001/jama.1997.03550230051036)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.1997.03550230051036&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=9403421&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1997YK98700036&link_type=ISI) 

18. Stiell IG, Clement CM, Grimshaw J, et al. Implementation of the Canadian C-Spine Rule: prospective 12 centre cluster randomised trial. BMJ 2009;339:b4146. [doi:10.1136/bmj.b4146](http://dx.doi.org/10.1136/bmj.b4146)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1136/bmj.b4146&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=19875425&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 

19. Cameron C, Naylor CD. No impact from active dissemination of the Ottawa Ankle Rules: further evidence of the need for local implementation of practice guidelines. CMAJ 1999;160:1165–8.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMDoiMTYwLzgvMTE2NSI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi82LzMvZTAwOTk1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

20. Boutis K, Grootendorst P, Willan A, et al. Effect of the Low Risk Ankle Rule on the frequency of radiography in children with ankle injuries. CMAJ 2013;185:E731–8. [doi:10.1503/cmaj.122050](http://dx.doi.org/10.1503/cmaj.122050)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTg1LzE1L0U3MzEiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi8zL2UwMDk5NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

21. Pozen MW, D'Agostino RB, Selker HP, et al. A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. A prospective multicenter clinical trial. N Engl J Med 1984;310:1273–8. [doi:10.1056/NEJM198405173102001](http://dx.doi.org/10.1056/NEJM198405173102001)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1056/NEJM198405173102001&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=6371525&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1984SR28000001&link_type=ISI) 

22. Kline JA, Zeitouni RA, Hernandez-Nino J, et al. Randomized trial of computerized quantitative pretest probability in low-risk chest pain patients: effect on safety and resource use. Ann Emerg Med 2009;53:727–35.e1. [doi:10.1016/j.annemergmed.2008.09.034](http://dx.doi.org/10.1016/j.annemergmed.2008.09.034)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.annemergmed.2008.09.034&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=19135281&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 

23. Hall LM, Jung RT, Leese GP. Controlled trial of effect of documented cardiovascular risk scores on prescribing. BMJ 2003;326:251–2. [doi:10.1136/bmj.326.7383.251](http://dx.doi.org/10.1136/bmj.326.7383.251)
    
    [FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjYvNzM4My8yNTEiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi8zL2UwMDk5NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

24. Hanon O, Franconi G, Mourad JJ, et al. [The estimation of cardiovascular risk in hypertensive patients is not modified by management of the hypertension]. Arch Mal Coeur Vaiss 2000;93:943–7.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=10989734&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000089059200009&link_type=ISI) 

25. Grover SA, Lowensteyn I, Joseph L, et al. Discussing coronary risk with patients to improve blood pressure treatment: secondary results from the CHECK-UP study. J Gen Intern Med 2009;24:33–9. [doi:10.1007/s11606-008-0825-4](http://dx.doi.org/10.1007/s11606-008-0825-4)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1007/s11606-008-0825-4&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=18937013&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000261969900006&link_type=ISI) 

26. Grover SA, Lowensteyn I, Joseph L, et al. Patient knowledge of coronary risk profile improves the effectiveness of dyslipidemia therapy: the CHECK-UP study: a randomized controlled trial. Arch Intern Med 2007;167:2296–303. [doi:10.1001/archinte.167.21.2296](http://dx.doi.org/10.1001/archinte.167.21.2296)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/archinte.167.21.2296&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=18039987&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000251217100004&link_type=ISI) 

27. Persell SD, Lloyd-Jones DM, Friesema EM, et al. Electronic health record-based patient identification and individualized mailed outreach for primary cardiovascular disease prevention: a cluster randomized trial. J Gen Intern Med 2013;28:554–60. [doi:10.1007/s11606-012-2268-1](http://dx.doi.org/10.1007/s11606-012-2268-1)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1007/s11606-012-2268-1&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=23143672&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 

28. McIsaac WJ, Goel V. Effect of an explicit decision-support tool on decisions to prescribe antibiotics for sore throat. Med Decis Making 1998;18:220–8.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=9566455&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000073022700010&link_type=ISI) 

29. McIsaac WJ, Goel V, To T, et al. Effect on antibiotic prescribing of repeated clinical prompts to use a sore throat score: lessons from a failed community intervention study. J Fam Pract 2002;51:339–44.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=11978257&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000174879500009&link_type=ISI) 

30. Worrall G, Hutchinson J, Sherman G, et al. Diagnosing streptococcal sore throat in adults: randomized controlled trial of in-office aids. Can Fam Physician 2007;53:666–71.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiY2ZwIjtzOjU6InJlc2lkIjtzOjg6IjUzLzQvNjY2IjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpvcGVuLzYvMy9lMDA5OTU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

31. Little P, Hobbs FD, Moore M, et al. Clinical score and rapid antigen detection test to guide antibiotic use for sore throats: randomised controlled trial of PRISM (primary care streptococcal management). BMJ 2013;347:f5806. [doi:10.1136/bmj.f5806](http://dx.doi.org/10.1136/bmj.f5806)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDcvb2N0MTBfMy9mNTgwNiI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi82LzMvZTAwOTk1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

32. McGinn TG, McCullagh L, Kannry J, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med 2013;173:1584–91. [doi:10.1001/jamainternmed.2013.8980](http://dx.doi.org/10.1001/jamainternmed.2013.8980)
    
    

33. Stiell IG, Clement CM, Grimshaw JM, et al. A prospective cluster-randomized trial to implement the Canadian CT Head Rule in emergency departments. CMAJ 2010;182:1527–32. [doi:10.1503/cmaj.091974](http://dx.doi.org/10.1503/cmaj.091974)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czoxMToiMTgyLzE0LzE1MjciO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi8zL2UwMDk5NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

34. McIsaac WJ, White D, Tannenbaum D, et al. A clinical score to reduce unnecessary antibiotic use in patients with sore throat. CMAJ 1998;158:75–83.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiY21haiI7czo1OiJyZXNpZCI7czo4OiIxNTgvMS83NSI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi82LzMvZTAwOTk1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

35. Stiell I, Wells G, Laupacis A, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre Ankle Rule Study Group. BMJ 1995;311:594–7. [doi:10.1136/bmj.311.7005.594](http://dx.doi.org/10.1136/bmj.311.7005.594)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTEvNzAwNS81OTQiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi8zL2UwMDk5NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

36. Bessen T, Clark R, Shakib S, et al. A multifaceted strategy for implementation of the Ottawa ankle rules in two emergency departments. BMJ 2009;339:b3056. [doi:10.1136/bmj.b3056](http://dx.doi.org/10.1136/bmj.b3056)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzkvYXVnMTJfMy9iMzA1NiI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi82LzMvZTAwOTk1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

37. Kerr D, Bradshaw L, Kelly AM. Implementation of the Canadian C-spine rule reduces cervical spine x-ray rate for alert patients with potential neck injury. J Emerg Med 2005;28:127–31. [doi:10.1016/j.jemermed.2004.08.016](http://dx.doi.org/10.1016/j.jemermed.2004.08.016)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jemermed.2004.08.016&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=15707805&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000227079000002&link_type=ISI) 

38. Stanley AJ, Ashley D, Dalton HR, et al. Outpatient management of patients with low-risk upper-gastrointestinal haemorrhage: multicentre validation and prospective evaluation. Lancet 2009;373:42–7. [doi:10.1016/S0140-6736(08)61769-9](http://dx.doi.org/10.1016/S0140-6736(08)61769-9)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(08)61769-9&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=19091393&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000262183800030&link_type=ISI) 

39. Sultan HY, Boyle A, Pereira M, et al. Application of the Canadian CT head rules in managing minor head injuries in a UK emergency department: implications for the implementation of the NICE guidelines. Emerg Med J 2004;21:420–5. [doi:10.1136/emj.2003.011353](http://dx.doi.org/10.1136/emj.2003.011353)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZW1lcm1lZCI7czo1OiJyZXNpZCI7czo4OiIyMS80LzQyMCI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi82LzMvZTAwOTk1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

40. Karlin-Zysman C, Zeitoun N, Belletti L, et al. Struggling to bring clinical prediction rules to the point of care: missed opportunities to impact patient care. J Comp Eff Res 2012;1:421–9. [doi:10.2217/cer.12.51](http://dx.doi.org/10.2217/cer.12.51)
    
    

41. Maguire JL, Kulik DM, Laupacis A, et al. Clinical prediction rules for children: a systematic review. Pediatrics 2011;128:e666–77. [doi:10.1542/peds.2011-0043](http://dx.doi.org/10.1542/peds.2011-0043)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjEyOC8zL2U2NjYiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi8zL2UwMDk5NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

42. Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med 2014;11:e1001744. [doi:10.1371/journal.pmed.1001744](http://dx.doi.org/10.1371/journal.pmed.1001744)
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1371/journal.pmed.1001744&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=25314315&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 

43. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. J Clin Epidemiol 2015;68: 134–43. [doi:10.1016/j.jclinepi.2014.11.010](http://dx.doi.org/10.1016/j.jclinepi.2014.11.010)
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=25579640&link_type=MED&atom=%2Fbmjopen%2F6%2F3%2Fe009957.atom) 

44. Pluddemann A, Wallace E, Bankhead C, et al. Clinical prediction rules in practice: review of clinical guidelines and survey of GPs. Br J Gen Pract 2014;64:e233–42. [doi:10.3399/bjgp14X677860](http://dx.doi.org/10.3399/bjgp14X677860)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMToiNjQvNjIxL2UyMzMiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNi8zL2UwMDk5NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

45. O'Rourke N. Review of Referral Patterns and Triage Processes in Symptomatic Breast Units. A hospital perspective. Dublin: National Cancer Control Programme; 2012.
    
    

46. Galvin R, Joyce D, Downey E, et al. Development and validation of a clinical prediction rule to identify suspected breast cancer: a prospective cohort study. BMC cancer 2014;14:743.
    
    

47. Framingham Heart Study. 2014. [https://www.framinghamheartstudy.org/risk-functions/coronary-heart-disease/10-year-risk.php](https://www.framinghamheartstudy.org/risk-functions/coronary-heart-disease/10-year-risk.php) (accessed 15 Jun 2015).
    
    

48. Hippisley-Cox J, Coupland C, Robson J, et al. Derivation, validation, and evaluation of a new QRISK model to estimate lifetime risk of cardiovascular disease: cohort study using QResearch database. BMJ 2010;341:c6624. [doi:10.1136/bmj.c6624](http://dx.doi.org/10.1136/bmj.c6624)
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDEvZGVjMDlfMS9jNjYyNCI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi82LzMvZTAwOTk1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=)