Finding order in heterogeneity: types of quality-improvement intervention publications
======================================================================================

* L V Rubenstein
* S Hempel
* M M Farmer
* S M Asch
* E M Yano
* D Dougherty
* P W Shekelle

## Abstract

**Background:** Stakeholders in quality improvement agree on the need for augmenting and synthesising the scientific literature supporting it. The diversity of perspectives, approaches, and contexts critical to advancing quality improvement science, however, creates challenges. The paper explores the heterogeneity in clinical quality improvement intervention (QII) publications.

**Methods:** A preliminary classification framework was developed for QII articles, aiming for categories homogeneous enough to support coherent scientific discussion on QII reporting standards and facilitate systematic review. QII experts were asked to identify articles important to QII science. The framework was tested and revised by applying it to the article set. The final framework screened articles into (1) empirical literature on development and testing of QIIs; (2) QII stories, theories, and frameworks; (3) QII literature syntheses and meta-analyses; or (4) development and testing of QII-related tools. To achieve homogeneity, category (1) required division into (1a) development of QIIs; 1(b) history, documentation, or description of QIIs; or (1c) success, effectiveness or impact of QIIs.

**Results:** By discussing unique issues and established standards relevant to each category, QII stakeholders can advance QII practice and science, including the scope and conduct of systematic literature reviews.

Quality of care in the United States is of uneven quality and often even unsafe.1 Clinical quality improvement intervention (QII) practitioners and evaluators seek to improve this situation by engaging organisations and their providers in developing and/or implementing better approaches to care delivery. In an era of cost containment, QIIs that aim to advance science as well as local care quality are urgently needed. Yet both publications, medicine’s mainstay for advancing science, and the synthesis of published work through systematic review have been problematic for the QII field. We aimed to develop a framework that recognised the diversity of types of articles essential to understanding and replicating QIIs, and that could be used for screening QII literature into more homogeneous categories for literature evaluation and synthesis.

An important challenge confronting stakeholders in promoting QII publication and synthesis has been the lack of agreed-upon standards for evaluating QII research. While agreement on standards is developing,23 there is as yet no method for systematically matching QII articles to relevant standards. In addition, there may be important types of QII articles that are excluded from most systematic reviews, yet could shed light on critical QII issues. To gain a comprehensive view of important article types, we asked experts to identify articles they considered important to advancing QII science. We then refined and tested our framework based on the resulting article set.

QIIs seek to improve the quality of care delivered by target organisations or organisational units such as practices, intensive care units, or community groups. QIIs can include, for example, changes in internal or external organisational policy, staffing, resources or any of the components of the chronic illness care model, such as links to external resources, decision support, informatics, care design or self-management support. In the work presented here, we focused on QIIs directed at improving evidence-based clinical practices such as those identified in guidelines. QIIs address both the method(s) for encouraging change (eg, continuous quality improvement, expert consultation or reimbursement changes) and the intended changes themselves. QIIs change structure or organisation in order to charge the process and outcomes of care. By nature, QIIs are carried out in diverse, often complex settings.4–13 Documentation and analysis of, for example, QII settings, encouragement methods, and direct and indirect outcomes may require a variety of approaches.

In addition, many QIIs employ theories, designs and methods that have not been extensively used in the clinical literature, with its focus on randomised controlled trials. Many of the approaches used are also new to classical health-services research, which relies heavily on epidemiologically based methodologies. Healthy development of the QII field may require studies focused on theories, terminology development, intervention development and hypothesis generation to a greater extent than traditional clinical or health services research. Recognising diverse types of studies and their goals, in addition to traditional hypothesis testing studies, may encourage more rapid, efficient progress toward effective QIIs. Our classification framework aims to highlight this diversity, while sorting QII publications into categories homogeneous enough to support scientific discussion of methodological and reporting standards potentially applicable to each category.

We report here on the development and preliminary testing of our classification framework. We assessed whether reviewers could apply the framework to screen articles suggested by experts into categories, and whether it comprehensively classified the full expert-generated article set.

## METHODS

For this project, we approached experts with extensive exposure to scientific approaches to clinical quality improvement who were part of a planning group for a research and evaluation designs and methods conference in 2005.14 The conference was directed at a broad group of about 100 stakeholders including journal editors, researchers, funders, and lay and professional quality-improvement practitioners and advocates. The planning group experts were health services and public-health researchers, many of whom had specific programmatic responsibility for developing QII within their organisations, that is the Agency for Healthcare Quality and Research, the Centers for Disease Control, the Veterans Administration, the National Institutes for Health and the Robert Wood Johnson Foundation. We asked them to identify examples of publications important to advancing QII science. We sought publications involving either healthcare or community organisations or organisational units, and that concerned the implementation of consensus guidelines or evidence-based clinical practices for a population of patients.

Experts suggested articles during regularly scheduled conference planning calls, one face-to-face planning meeting, or by email over a 5-month period beginning in 9/04, and conducted a final review of our article list in 2/05. The core planning group initially included eight people from participating organisations but gradually expanded to 22 plus five additional intermittent consultants. Of the final 27 total, 12 either contributed articles or commented on the final article list.

We developed a classification framework using an iterative process. Our goal was to achieve categories that were sensitive (included a high proportion of the expert-identified articles), could be reliably applied by reviewers and had face validity as a starting-point for applying the right methodological standards to identified articles. Groups of two to five reviewers active in QII research (three senior physician health services researchers (LR, SA, PS), one senior PhD health services researcher (EY), and one junior PhD health services researcher (MF) scrutinised identified publications assessing similarities and differences between publications. The evolving categories were further discussed regarding their potential for specifying types of articles to which coherent methodological norms or standards could be applied. We applied categories through a decision tree operationalised as a standardised screening form, and sequentially improved the form through expanding cycles of review.

Two reviewers (LR and MF) independently applied the screening form to the full set of identified publications. Disagreements were resolved through consensus. As a quality check, three further reviewers independently subjected selected articles to the screening form and compared results. Two additional authors (SH and DD) then reviewed the final article set and categories to check final article assignment.

## RESULTS

The set of 80 publications identified by the expert panel is documented in the Appendix (online). The iterative review process revealed these publication types: (I) Empirical literature on development and testing of QIIs; (II) QII stories, theories, and frameworks; (III) QII literature synthesis and meta-analysis; (IV) Development and testing of QII-related tools, and a miscellaneous category for books and unclassifiable articles. Table 1 provides an overview of the general type of publication, the underlying goal, a broad definition and the number of identified studies within the expert selection. The table also summarises the resulting methodological issues within the framework of QII research that each type of publication may represent or may be able to address.

View this table:
[Table 1](http://qualitysafety.bmj.com/content/17/6/403/T1)

Table 1 Categories of quality-improvement intervention (QII) research

Twenty-five articles15–39 reported data collected to develop or test QIIs (Category I). These studies aimed to improve quality of care provided by a specific organisational unit or units, practice or community for a defined population.

Thirty articles4–940–63 discussed theories, frameworks or “stories” relevant to QIIs (Category II). This type included articles that did *not* report on formal qualitative or quantitative data collection carried out for the purpose of the subject article, other than expert panel-type data, and that were not formal literature syntheses or meta-analyses.

Ten articles64–73 were formal literature syntheses or meta-analyses focused on QII research (Category III).

Nine articles74–82 dealt with QII tools (Category IV). Tools are tests or methods to assess and document any QII. These publications report data concerning the development, testing or refinement of QII methods or tools in naturalistic settings (eg, sensitivity, specificity, reliability, validity, usefulness, acceptability).

Four of the publications11–1383 suggested by the experts were books on quality and evaluation methods, and could not be classified in the four main categories. Two publications8485 focused on consent procedures and a Cochrane group description of non-randomised methods.

The review process revealed that Group I, the empirical literature on development and testing of QIIs, should be further distinguished into the subcategories: (Ia) Development of QIIs; (Ib) History, documentation, or description of QIIs; (Ic) Success, effectiveness or impact of QIIs. Table 2 outlines the goals, definitions and number of identified studies, and summarises further methodological issues and conceptual considerations.

View this table:
[Table 2](http://qualitysafety.bmj.com/content/17/6/403/T2)

Table 2 Group I empirical studies subtypes

Eight of the 25 category I articles15–22 focused on the preintervention phase and reported data collected for the purpose of developing QIIs (Category Ia).

Two articles focused on the postintervention phase (Category Ib) but did not report empirical data on the success, effectiveness or impacts of a single QII study and were therefore deemed to represent a distinct subtype.2324

Fifteen articles reported data on QII success, effectiveness or impacts (Category Ic).25–39 This subtype of publication included randomised trials as well as a variety of more naturalistic designs.

## DISCUSSION

This project aimed to explore the heterogeneity of QII publications by determining what types of articles experts would identify as germane to the field of QII research. We discovered a rich and heterogeneous literature. Within this heterogeneity, however, we identified several relatively homogeneous types of publications identified by experts as important to clinical QII research. These types of publications covered nearly all suggested articles, had face validity and could be reliably identified by independent reviewers using a decision tree. This broad categorisation appears suitable for promoting coherent discussion around standards for conduct and publication directed at any one of the specific types, and as the basis for initial screening of articles for systematic reviews.

Three of the categories included articles with a focus on reporting data on QIIs, while one included articles without such a focus. The first type encompassed a diverse set of data-based articles on QIIs spanning all phases from development through evaluation. Within this category, a number of important sources of methodological heterogeneity can be identified that merit further analysis and discussion. A challenging but typical aspect of publications in this category was that the full set of information about a particular study was often presented in multiple and methodologically diverse articles.36 Judgements of data-based QII article quality thus may require, for example, consideration of qualitative research standards and randomised trial standards as applicable across relevant article sets. Another challenging aspect of articles in this category was based on the multiyear, multiphase, evolutionary nature of many QIIs, resulting in different study goals at different phases of scientific learning.

We distinguished three subcategories of data-based QIIs that took account of different study goals. Knowledge based on each subcategory appeared crucial for understanding the QII’s context as required by realistic evaluation86 frameworks. Subtype Ia studies were concerned with the development of QIIs. These raised the issue of the role of a systematic development process as part of QII projects in general. Does a development process that begins with evidence review, plan–do–study–act cycles87 and systematic engagement of stakeholders in design produce a higher-quality QII? Subtype Ib studies addressed the history, documentation or description of QIIs. While the other subtypes represented studies also commonly found in other fields,88 subtype Ib studies appeared more distinct. These articles did not report new data on whether the QII achieved the intended changes in clinical process or outcomes, but reported on, for example, challenges or critical lessons learnt across QII stages or a series of QIIs. While no articles describing an individual QII were suggested by our experts, we envision this category as including articles that focus, for example, on data showing whether an intervention was implemented as planned. In addition to their importance for understanding QII evaluation results, such articles may set the stage for subsequent rigorous approaches such as preplanned meta-analyses across a set of implementation studies or other kinds of crosscutting research aimed at understanding the impacts of whole initiatives.

Type Ic studies evaluated the success, effectiveness or impact of QIIs—a type of article one would immediately think of as integral to QII research, yet one that represented only a minority among articles our experts considered to be important. In addition to reflecting the diversity of important work in the QII field, this may reflect a relative dearth of high-quality empirical publications evaluating QIIs. The lack of standard expectations for what should be included in evaluating and reporting on QIIs may make it particularly difficult for authors and reviewers to achieve common ground on papers, further diminishing the number of QIIs entering the literature. The many existing quality-improvement initiatives in clinical practice are thus not currently well reflected in the literature.2 Further exploration of studies in this category may yield additional insights about how researchers are currently handling the challenges inherent in QII research, and eventually greater consensus about which approaches fit which purposes. Although the verdict on suitable study designs for evaluating QII success, effectiveness or impact may still be out, many reporting and methodological standards are available.38990 We used an article classification decision tree to assign articles to our categories. Existing standards could be applied to appropriate articles or article sections based on further development of the decision tree past the initial screening stage reported on here.

The second group (Category II) focused on articles that did not focus on data. These included stories, theories and frameworks related to QIIs. Clearly, theory is essential to the development of a deeper understanding of the mechanisms of action of QIIs, and hence to our ability to predict the usefulness of particular improvement approaches under particular conditions. Deming, for example, recognised the importance of taking account of “profound knowledge” such as psychological theories in quality improvement work.91 Our reasoning on understanding causality in QII work, where context is important, requires scrutiny.92 Theoretical development in the literature sample focused both on developing theories about QIIs and on applying existing social science theory to QIIs. Articles on frameworks addressed the underlying concepts used in implementing and evaluating QIIs and were used to generate, organise or promote hypotheses. Articles on stories provided a chronological description of what happened as part of implementing a QII, but were not based on systematic data collection. Learning theory and practical application demonstrate the power of stories in communicating about quality improvement.93–95 We did not attempt to subdivide Category II articles; further discussion of these articles, however, might have been fruitful. For example, articles that “tell the story” of a QII might be considered to be of a higher quality if they represented first-hand knowledge and/or were reviewed by multiple QII participants. Conceptual frameworks might be rated as of a higher quality if they have undergone expert panel review. In addition, subdivisions of Category II might consider, for example Tilly’s four categories of explanatory reasons (conventions, formulas, stories and technical accounts),91 or other theoretical frameworks. Focusing on this category of publications will encourage literature synthesis that includes rather than excludes important conceptual articles and that thus enables systematic development of the theoretical basis for quality improvement.

The third type of article focused on developing the QII evidence base through data from literature syntheses and meta-analyses. The importance of accumulating knowledge across studies about quality-improvement interventions is self-evident. Discussions of methodological issues in this category, such as how best to assess multicomponent and organisational interventions through meta-analysis, are likely to be fruitful. Methodological quality standards that can be applied by reviews dealing specifically with QIIs, such as those in progress through the SQUIRE group initiative,2 will be critical for developing this type of research.

Finally, the fourth category may seem surprising as a QII research category at first glance, as this category dealt with data on the development or testing of QII tools in routine care or other naturalistic settings. These articles often used classical methods of measure development such as verification, validation, user acceptability or human-factors testing.96 On further investigation, however, it is apparent that a focus on QII measurement is critical. Measures developed and tested only within a research context may not be valid or reliable when applied by usual personnel in actual care sites, and may be impractical or unfeasible. On the other hand, measures developed solely for the purposes of a QII may not have undergone testing at all, and may be highly misleading. Whatever their origins, the use of measures that have not been specifically tested in naturalistic settings can have serious negative organisational and clinical consequences in a QII context, resulting in erroneous clinical decisions, excessive costs or invalid evaluation. Quality standards for this type of article would involve those related to instrument development, but with attention to issues related to the QII context. Measurement work in the context of QII development and evaluation should be a focus of further discussion and investigation.

This study has limitations. First, the article sample, while representative of the opinions of a diverse group of experts, was essentially a convenience sample, albeit larger than samples used in some prior published studies using similar methodology.397 Readers should therefore not infer that, for example, the proportions of articles in each category are representative of the full QII literature. Second, this article focuses on initial classification of articles into broad categories. A variety of social science9899 and QI2354868897 theorists have developed frameworks that may prove fruitful in understanding and improving QII research. In our initial empirically based article screening strategy, however, these theories, as well as other study dimensions such as outcomes would be considered at a later stage of review. Finally, this study sought articles whose goals were to produce organisational or structural change either in healthcare settings or in the community. While this aim excludes the large number of studies whose goal is to test the efficacy or effectiveness of an intervention independently of organisational context, we see this focus as productive in identifying articles directly aimed at improving the quality of routine healthcare and health promotion.

In conclusion, we found both heterogeneity and coherence in the QII literature we reviewed. By considering each important type of literature separately, and building on existing investigative approaches to understand and refine methods, QII research stakeholders can advance the practice and science in this field.

## Acknowledgments

The authors would like to thank the following individuals who helped guide the project as part of planning for the symposium, Expanding Research and Evaluation Designs to Improve the Science Base for Health Care and Public Health Quality Improvement: D Atkins, AHRQ; CT Orleans and L Leviton, RWJF; S Mercer and P Briss, CDC; L Fine, NIH; E DeVoto, NIH; J Francis, VA; LW Green, independent consultant; and B DeVinney, Independent Consultant. A summary of the symposium is available at [http://www.hsrd.research.va.gov/quality2005](http://www.hsrd.research.va.gov/quality2005).

## Footnotes

*   **Competing interests:** None.

*   **Funding:** Agency for Healthcare Research and Quality (AHRQ) under a contract to LVR (contract no HHSP233200400717P) funded by an Interagency Agreement between AHRQ and the Centers for Disease Control and Prevention.

*   **See Editorial, p 394**

## REFERENCES

1.  Agency for Healthcare Research and Quality. National Healthcare Disparities report. Rockville: Agency for Healthcare Research and Quality, 2006. [http://www.ahrq.gov/qual/qrdr07.htm](http://www.ahrq.gov/qual/qrdr07.htm) (accessed 14 Oct 2008).
    
    

2.  Davidoff F, Batalden P. Toward stronger evidence on quality improvement. Draft publication guidelines: the beginning of a consensus project. Qual Saf Health Care 2005;14:319–25 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjE0LzUvMzE5IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  Lohr KN. Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care 2004;16:9–18 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiaW50cWhjIjtzOjU6InJlc2lkIjtzOjY6IjE2LzEvOSI7czo0OiJhdG9tIjtzOjE4OiIvcWhjLzE3LzYvNDAzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

4.  Speroff T, O’Connor GT. Study designs for PDSA quality improvement research. Qual Manag Health Care 2004;13:17–32 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00019514-200401000-00002&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14976904&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

5.  Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003;12:458–64 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjEyLzYvNDU4IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

6.  Biglan A, Ary D, Wagenaar AC. The value of interrupted time-series experiments for community intervention research. Prev Sci 2000;1:31–49 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1023/A:1010024016308&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11507793&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

7.  Speroff T, James BC, Nelson EC, et al. Guidelines for appraisal and publication of PDSA quality improvement. Qual Manag Health Care 2004;13:33–9 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00019514-200401000-00003&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14976905&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

8.  Moss F, Thompson R. A new structure for quality improvement reports. Qual Health Care 1999;8:76.
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000081164800003&link_type=ISI) 

9.  Des Jarlais DC, Lyles C, Crepaz N. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health 2004;94:361–6 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.2105/AJPH.94.3.361&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14998794&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000189317400004&link_type=ISI) 

10. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637–9 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.1996.03540080059030&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=8773637&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1996VC85300031&link_type=ISI) 

11. Patton MQ. Utilization-focused evaluation. Beverly Hills: Sage, 1986.
    
    

12. Shadish WR, Cook TD, Campbell D. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin, 2001.
    
    

13. Yin RK. Case study research design and methods.
    
    

14. Health Services Research and Development. Expanding Research and Evaluation Designs to Improve the Science Base for Health Care and Public Health Quality Improvement, 13 September 2005–15 September 2005. [http://www.hsrd.research.va.gov/quality2005](http://www.hsrd.research.va.gov/quality2005) (accessed 8 Oct 2008).
    
    

15. Murff HJ, Gandhi TK, Karson AK, et al. Primary care physician attitudes concerning follow-up of abnormal test results and ambulatory decision support systems. Int J Med Inform 2003;71:137–49 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S1386-5056(03)00133-3&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14519406&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000186023800005&link_type=ISI) 

16. Ozer EM, Adams SH, Gardner LR, et al. Provider self-efficacy and the screening of adolescents for risky health behaviors. J Adolesc Health 2004;35:101–7 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15261638&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000222880400005&link_type=ISI) 

17. Vargas PA, Simpson PM, Gary Wheeler J, et al. Characteristics of children with asthma who are enrolled in a Head Start program. J Allergy Clin Immunol 2004;114:499–504 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.jaci.2004.05.025&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15356547&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000223799600004&link_type=ISI) 

18. Leviton LC, Baker S, Hassol A, et al. An exploration of opinion and practice patterns affecting low use of antenatal corticosteroids. Am J Obstet Gynecol 1995;173:312–6 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/0002-9378(95)90220-1&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=7631711&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1995RL64600061&link_type=ISI) 

19. Phillips LS, Hertzberg VS, Cook CB, et al. The Improving Primary Care of African Americans with Diabetes (IPCAAD) project: rationale and design. Control Clin Trials 2002;23:554–69 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0197-2456(02)00230-1&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12392871&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000178326200009&link_type=ISI) 

20. Cretin S, Shortell SM, Keeler EB. An evaluation of collaborative interventions to improve chronic illness care. Framework and study design. Eval Rev 2004;28:28–51 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BlcngiO3M6NToicmVzaWQiO3M6NzoiMjgvMS8yOCI7czo0OiJhdG9tIjtzOjE4OiIvcWhjLzE3LzYvNDAzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

21. Horbar JD, Carpenter JH, Buzas J, et al. Timing of initial surfactant treatment for infants 23 to 29 weeks’ gestation: is routine practice evidence based? Pediatrics 2004;113:1593–602 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6InBlZGlhdHJpY3MiO3M6NToicmVzaWQiO3M6MTA6IjExMy82LzE1OTMiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

22. Frayne SM, Skinner KM, Sullivan LM, et al. Sexual assault while in the military: violence as a predictor of cardiac risk? Violence Vict 2003;18:219–25 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1891/vivi.2003.18.2.219&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12816405&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

23. Finkelstein JA, Lozano P, Streiff KA, et al. Clinical effectiveness research in managed-care systems: lessons from the Pediatric Asthma Care PORT. Patient Outcomes Research Team. Health Serv Res 2002;37:775–89 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/1475-6773.00048&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12132605&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000176617500013&link_type=ISI) 

24. Feifer C, Fifield J, Ornstein S, et al. From research to daily clinical practice: what are the challenges in “translation”? Jt Comm J Qual Saf 2004;30:235–45 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15154315&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

25. Jones KR, Fink R, Pepper G, et al. Improving nursing home staff knowledge and attitudes about pain. Gerontologist 2004;44:469–78 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiZ2Vyb250IjtzOjU6InJlc2lkIjtzOjg6IjQ0LzQvNDY5IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

26. Lozano P, Finkelstein JA, Carey VJ, et al. A multisite randomized trial of the effects of physician education and organizational change in chronic-asthma care: health outcomes of the Pediatric Asthma Care Patient Outcomes Research Team II Study. Arch Pediatr Adolesc Med 2004;158:875–83 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/archpedi.158.9.875&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15351753&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000223750000005&link_type=ISI) 

27. Margolis PA, Lannon CM, Stuart JM, et al. Practice based education to improve delivery systems for prevention in primary care: randomised trial. BMJ 2004;328:388.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjgvNzQzNi8zODgiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

28. Landon BE, Wilson IB, McInnes K, et al. Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study. Ann Intern Med 2004;140:887–96 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-140-11-200406010-00010&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15172903&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000221680600005&link_type=ISI) 

29. Chin MH, Cook S, Drum ML, et al. Improving diabetes care in midwest community health centers with the health disparities collaborative. Diabetes Care 2004;27:2–8 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiZGlhY2FyZSI7czo1OiJyZXNpZCI7czo2OiIyNy8xLzIiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

30. Margolis PA, Stevens R, Bordley WC, et al. From concept to application: the impact of a community-wide intervention to improve the delivery of preventive services to children. Pediatrics 2001;108:E42.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1542/peds.108.3.e42&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11533360&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

31. Feifer C, Ornstein SM. Strategies for increasing adherence to clinical guidelines and improving patient outcomes in small primary care practices. Jt Comm J Qual Saf 2004;30:432–41 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15357133&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

32. Solberg LI, Kottke TE, Brekke ML, et al. Failure of a continuous quality improvement intervention to increase the delivery of preventive services. A randomized trial. Eff Clin Pract 2000;3:105–15 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11182958&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

33. Leviton LC, Goldenberg RL, Baker CS, et al. Methods to encourage the use of antenatal corticosteroid therapy for fetal maturation: a randomized controlled trial. JAMA 1999;281:46–52 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.281.1.46&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=9892450&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000077829300034&link_type=ISI) 

34. Beyer M, Gerlach FM, Flies U, et al. The development of quality circles/peer review groups as a method of quality improvement in Europe. Results of a survey in 26 European countries. Fam Pract 2003;20:443–51 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiZmFtcHJhY3QiO3M6NToicmVzaWQiO3M6ODoiMjAvNC80NDMiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

35. Shafer MA, Tebb KP, Pantell RH, et al. Effect of a clinical practice improvement intervention on Chlamydial screening among adolescent girls. JAMA 2002;288:2846–52 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.288.22.2846&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12472326&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000179732600029&link_type=ISI) 

36. Ornstein S, Jenkins RG, Nietert PJ, et al. A multimethod quality improvement intervention to improve preventive cardiovascular care: a cluster randomized trial. Ann Intern Med 2004;141:523–32 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-141-7-200410050-00008&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15466769&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000224467800004&link_type=ISI) 

37. Schoenbaum M, Unutzer J, Sherbourne C, et al. Cost-effectiveness of practice-initiated quality improvement for depression: results of a randomized controlled trial. JAMA 2001;286:1325–30 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.286.11.1325&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11560537&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000171024500025&link_type=ISI) 

38. Horbar JD, Carpenter JH, Buzas J, et al. Collaborative quality improvement to promote evidence based surfactant for preterm infants: a cluster randomised trial. BMJ 2004;329:1004.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjkvNzQ3My8xMDA0IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

39. Solberg LI, Kottke TE, Brekke ML, Magnan S. Improving prevention is difficult. Eff Clin Pract 2000;3:153–5 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11182966&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

40. Hanney SR, Gonzalez-Block MA, Buxton MJ, et al. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst 2003;1:2.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1186/1478-4505-1-2&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12646071&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

41. Davis P, Howden-Chapman P. Translating research findings into health policy. Soc Sci Med 1996;43:865–72 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/0277-9536(96)00130-X&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=8870150&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1996VD07800028&link_type=ISI) 

42. Berridge V, Stanton J. Science and policy: historical insights. Soc Sci Med 1999;49:1133–8 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0277-9536(99)00155-0&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10501636&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000082198400001&link_type=ISI) 

43. Sung NS, Crowley WF, Jr, Genel M, et al. Central challenges facing the national clinical research enterprise. JAMA 2003;289:1278–87 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.289.10.1278&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12633190&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000181466500030&link_type=ISI) 

44. Rosenberg RN. Translating biomedical research to the bedside: a national crisis and a call to action. JAMA 2003;289:1305–6 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.289.10.1305&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12633194&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000181466500034&link_type=ISI) 

45. Grol R, Grimshaw J. Evidence-based implementation of evidence-based medicine. Jt Comm J Qual Improv 1999;25:503–13 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10522231&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

46. Wood M, Ferlie E, Fitzgerald L. Achieving clinical behaviour change: a case of becoming indeterminate. Soc Sci Med 1998;47:1729–38 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0277-9536(98)00250-0&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=9877343&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000077452200010&link_type=ISI) 

47. Kitson A. Recognising relationships: reflections on evidence-based practice. Nurs Inq 2002;9:179–86 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1046/j.1440-1800.2002.00151.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12199882&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

48. Lomas J. Using “linkage and exchange” to move research into policy at a Canadian foundation. Health Aff (Millwood) 2000;19:236–40 .
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo5OiJoZWFsdGhhZmYiO3M6NToicmVzaWQiO3M6ODoiMTkvMy8yMzYiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

49. Berwick DM. Disseminating innovations in health care. JAMA 2003;289:1969–75 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.289.15.1969&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12697800&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000182253900030&link_type=ISI) 

50. Eccles M, Grimshaw J, Campbell M, et al. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 2003;12:47–52 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjEyLzEvNDciO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

51. Leape LL, Berwick DM, Bates DW. What practices will most improve safety? Evidence-based medicine meets patient safety. JAMA 2002;288:501–7 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.288.4.501&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12132984&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000176955400036&link_type=ISI) 

52. Wandersman A. Community science: bridging the gap between science and practice with community-centered models. Am J Community Psychol 2003;31:227–42 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1023/A:1023954503247&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12866681&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000183206100006&link_type=ISI) 

53. Olsson JA, Ovretveit J, Kammerlind P. Developing and testing a model to predict outcomes of organizational change. Qual Manag Health Care 2003;12:240–9 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14603786&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

54. Ovretveit J, Bate P, Cleary P, et al. Quality collaboratives: lessons from research. Qual Saf Health Care 2002;11:345–51 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjExLzQvMzQ1IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

55. Centers for Disease Control and Prevention. CDC’s 7 Healthcare Safety Challenges, 23 August 2001. [http://www.cdc.gov/ncidod/dhqp/about\_challenges.html#](http://www.cdc.gov/ncidod/dhqp/about_challenges.html#) (accessed 8 Oct 2008).
    
    

56. Epping-Jordan JE, Pruitt SD, Bengoa R, et al. Improving the quality of health care for chronic conditions. Qual Saf Health Care 2004;13:299–305 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjEzLzQvMjk5IjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

57. Ornstein SM. Translating research into practice using electronic medical records the PPRNet-TRIP project: primary and secondary prevention of coronary heart disease and stroke. Top Health Inf Manage 2001;22:52–8 .
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11761792&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

58. Shojania K, McDonald K, Watcher R, et al. Closing the gap: a critical analysis of quality improvement strategies [dissertaion]. Lubbock: Texas Tech University, 1999.
    
    

59. Atkins D, Best D, Briss PA, et al. Grading quality of evidence and strength of recommendations. BMJ 2004;328:1490.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjgvNzQ1NC8xNDkwIjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

60. Wilson AR, Dowd BE, Krawlewski JE. Medical group practices patient safety research: measurements and data challenges. Minneapolis: Division of Health Services Research and Policy, 2004.
    
    

61. Glasgow RE, Bull SS. Making a difference with interactive technology: considerations in using and evaluating computerized aids for diabetes self-management education. Diabetes Spectrum 2001;14:99–106 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiZGlhc3BlY3QiO3M6NToicmVzaWQiO3M6NzoiMTQvMi85OSI7czo0OiJhdG9tIjtzOjE4OiIvcWhjLzE3LzYvNDAzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

62. Ophoven J. Bridging research and practice focus on effective dissemination strategies. Woodbury: The Crackleberry Group, 1997.
    
    

63. Freund D, Lave J, Clancy C, et al. Patient Outcomes Research Teams: contribution to outcomes and effectiveness research. Annu Rev Public Health 1999;20:337–59 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1146/annurev.publhealth.20.1.337&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10352862&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000080438800019&link_type=ISI) 

64. Farquhar CM, Stryer D, Slutsky J. Translating research into practice: the future ahead. Int J Qual Health Care 2002;14:233–49 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiaW50cWhjIjtzOjU6InJlc2lkIjtzOjg6IjE0LzMvMjMzIjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

65. Shortell SM, Bennett CL, Byck GR. Assessing the impact of continuous quality improvement on clinical practice: what it will take to accelerate progress. Milbank Q 1998;76:593–624, 510.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/1468-0009.00107&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=9879304&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000077653800004&link_type=ISI) 

66. Grol R. Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA 2001;286:2578–85 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.286.20.2578&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11722272&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000172385200032&link_type=ISI) 

67. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet 2003;362:1225–30 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(03)14546-1&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14568747&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000185924100026&link_type=ISI) 

68. Bero LA, Grilli R, Grimshaw JM, et al. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. BMJ 1998;317:465–8 .
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMTcvNzE1Ni80NjUiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

69. Grimshaw JM, Shirran L, Thomas R, et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001;39(8 Suppl 2):II2–45 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00005650-200108002-00002&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11583120&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000170313600002&link_type=ISI) 

70. Chang JT, Morton SC, Rubenstein LZ, et al. Interventions for the prevention of falls in older adults: systematic review and meta-analysis of randomised clinical trials. BMJ 2004;328:680.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjgvNzQ0MS82ODAiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

71. US Department of Health and Human Services. *National Healthcare Quality Report*. Rockville: Agency for Healthcare Research and Quality, 2003.
    
    

72. Kane RL, Johnson PE, Town RJ, et al. Economic incentives for preventive care. Evid Rep Technol Assess (Summ) 2004:1–7 .
    
    

73. Greco PJ, Eisenberg JM. Changing physicians’ practices. N Engl J Med 1993;329:1271–3 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJM199310213291714&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=8413397&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1993MB98800014&link_type=ISI) 

74. Gustafson DH, Sainfort F, Eichler M, et al. Developing and testing a model to predict outcomes of organizational change. Health Serv Res 2003;38:751–76 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/1475-6773.00143&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12785571&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000182793400016&link_type=ISI) 

75. Clark L, Jones K, Pennington K. Pain assessment practices with nursing home residents. West J Nurs Res 2004;26:733–50 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3B3am4iO3M6NToicmVzaWQiO3M6ODoiMjYvNy83MzMiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

76. Poon EG, Wang SJ, Gandhi TK, et al. Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform 2003;36:80–91 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/S1532-0464(03)00061-3&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=14552849&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000186072200008&link_type=ISI) 

77. Hofer TP, Asch SM, Hayward RA, et al. Profiling quality of care: Is there a role for peer review? BMC Health Serv Res 2004;4:9.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1186/1472-6963-4-9&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=15151701&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 

78. Solodky C, Chen H, Jones PK, et al. Patients as partners in clinical research: a proposal for applying quality improvement methods to patient care. Med Care 1998;36(8 Suppl):AS13–20 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00005650-199808001-00003&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=9708579&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000075295300003&link_type=ISI) 

79. The CAHPS Connection. ICSI: a collaboration to improve patients’ experiences. Rockville: Agency for Healthcare Research and Quality, 2004. [http://www.cahps.ahrq.gov/content/CAHPSConnection/CAHPSConnectionIndex.asp](http://www.cahps.ahrq.gov/content/CAHPSConnection/CAHPSConnectionIndex.asp) (accessed 14 Oct 2008).
    
    

80. Powe NR, Turner JA, Maklan CW, et al. Alternative methods for formal literature review and meta-analysis in AHCPR Patient Outcomes Research Teams. Agency for Health Care Policy and Research. Med Care 1994;32(7 Suppl):JS22–37 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00005650-199407001-00002&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=8028411&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1994NW56000003&link_type=ISI) 

81. Pbert L, Vuckovic N, Ockene JK, et al. Developing and testing new smoking measures for the Health Plan Employer Data and Information Set. Med Care 2003;41:550–9 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00005650-200304000-00010&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12665718&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000182006400010&link_type=ISI) 

82. Lorenz K. Validation of pain as a vital sign among veterans with advanced illness. Los Angeles: VA Greater Los Angeles Healthcare System.
    
    

83. Moen RD, Nolan TD, Provost LP. Improving quality through planned experimentation. New York: McGraw-Hill, 1991.
    
    

84. Reeves B. Cochrane Non-Randomised Studies Methods Group. [http://www/cochrane.dk/nrmg/module.htm](http://www/cochrane.dk/nrmg/module.htm) (accessed 13 Oct 2008).
    
    

85. Nelson K, Garcia RE, Brown J, et al. Do patient consent procedures affect participation rates in health services research? Med Care 2002;40:283–8 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00005650-200204000-00004&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=12021684&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000174712000004&link_type=ISI) 

86. Pawson R, Tilley N. Realistic evaluation. London: Sage, 1997.
    
    

87. Batalden PB, Davidoff F. What is “quality improvement” and how can it transform healthcare? Qual Saf Health Care 2007;16:2–3 .
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjY6IjE2LzEvMiI7czo0OiJhdG9tIjtzOjE4OiIvcWhjLzE3LzYvNDAzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

88. Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions to improve health. BMJ 2000;321:694–6 .
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjEvNzI2Mi82OTQiO3M6NDoiYXRvbSI7czoxODoiL3FoYy8xNy82LzQwMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

89. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:1–12 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/0197-2456(95)00134-4&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=8721797&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1996UA39800001&link_type=ISI) 

90. Juni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ 2001;323:42–6 .
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjExOiIzMjMvNzMwMy80MiI7czo0OiJhdG9tIjtzOjE4OiIvcWhjLzE3LzYvNDAzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

91. Walton M. The Deming management method. Letchworth: The Orbital Press.
    
    

92. Tilly C. Why? What happens when people give reasons...and why. Princeton: Princeton University Press, 2006.
    
    

93. Goodman A. Storytelling as best practice: how stories strengthen your organization. Los Angeles: A Goodman, 2004.
    
    

94. Greenhalgh T, Russell J, Swinglehurst D. Narrative methods in quality improvement research. Qual Saf Health Care 2005;14:443–9 .
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjE0LzYvNDQzIjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMTcvNi80MDMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

95. McCarthy D, Blumenthal D. Stories from the sharp end: case studies in safety improvement. Milbank Q 2006;84:165–200 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1468-0009.2006.00442.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=16529572&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000235893400007&link_type=ISI) 

96. Sawyer D. Do it by design. An introduction to human factors in medical devices. Rockville: US Department of Health and Human Services, Public Health Service, Food and Drug Administration, Center for Devices and Radiological Health, 1996.
    
    

97. Gartlehner G, Hansen RA, Nissman D, et al. A simple and valid tool distinguished efficacy from effectiveness studies. J Clin Epidemiol 2006;59:1040–8 .
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2006.01.011&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=16980143&link_type=MED&atom=%2Fqhc%2F17%2F6%2F403.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000241064500005&link_type=ISI) 

98. Flyveberg B. Making social science matter. Why social inquiry fails and how it can succeed again. Cambridge: Cambridge University Press, 2001.
    
    

99. 1.  Chelimsky E, 
    2.  Shadish W R
    
    , eds. Evaluation for the 21st century. A handbook. Thousand Oaks: Sage, 1997.