Improving MRCP PACES pass rates through the introduction of a regional multifaceted support framework

Problem

National Membership of Royal College of Physicians (MRCP) statistics for 2014–2015, indicate that only 67% of West Midlands’ core medical trainees (CMTs) had completed Practical Assessment of Clinical Examination Skills (PACES) by the time of their final Annual Review of Competence Progression (ARCP), in comparison with a national average of 78%.1 Moreover, local data for trainees within Coventry and Warwickshire demonstrated an even lower pass rate of 56.3%.

Through personal and peers’ experience of undertaking the MRCP PACES exam, the authors identified suboptimal support, resources and learning opportunities for PACES exam preparation locally.

Although a robust weekly teaching programme was operational at University Hospital Coventry and Warwickshire (UHCW), designed to cover key aspects of the CMTs' e-portfolio curriculum, there was no PACES-focused teaching framework. The MRCP exam series is a mandatory requirement for progression through core medical training onto specialty trainee registrar (ST) training. Delay in obtaining RCP membership results in delayed career progression, postponing trainees from securing higher specialist training posts, in turn causing variable vacancies in specialist doctors’ positions nationally, potentially affecting quality service provision.

This project aims to identify and remove barriers to effective ward-based learning by introduction of a robust PACES-focused teaching framework including a peer-to-peer mentoring scheme. It is believed that these interventions will improve trainee engagement with the MRCP exam series and improve local examination pass rates, while boosting doctors’ morale.

Background

The MRCP exam series is a prerequisite for all CMTs to progress onto higher-level training. The exam series consists of three parts: two written aspects (parts 1 and 2) and the PACES examination. National statistics demonstrate lower pass rates for trainees in the PACES examination, 53.8%, as compared with the two written examinations (56.8% and 75.8% for parts 1 and 2, respectively).2 Several well established online resources are available to aid preparation for the written exams, in the form of question banks and past papers.

However, given the practical nature of the PACES exam, exposure to appropriate patients in clinical practice and availability of structured feedback from senior colleagues is required. Although there are several national PACES courses offering this they are often expensive, places are quickly filled and dates prior to trainees’ examinations are sometimes unavailable. Furthermore, the majority of these courses are in London and other major cities, compelling a large cohort of candidates to embark on cumbersome travel and stay during the vital pre-examination period.

Preparation for PACES exam requires long-term, consistent trainee engagement with regular practice of examination skills. This helps trainees feel more prepared for the examination, and builds confidence in their role as a senior doctor in day-to-day working.3 This crucial link is often overlooked by trainees, who are superficially focused on passing their exams. This project aims to overcome this mindset, and provide a framework to allow structured preparation well in advance of exams.

Baseline measurement

Final ARCP outcome data for CMTs in Coventry and Warwickshire for 2014–2015 were analysed to assess the proportion of trainees who had successfully completed the MRCP exams. Of a total 16 final year CMTs, only 9 had completed MRCP PACES by their ARCP date giving a pass rate of 56.3%.

Baseline Likert Scale questionnaires (online supplementary file) were distributed to all CMTs in Coventry and Warwickshire to determine the current quality of the local PACES teaching. Likert Scales were used, as they are easy to understand and allow participants to offer positive, negative and neutral opinions. Furthermore, they allow for easy interpretation and graphical representation of quantitative data.4

Forty questionnaires were distributed with 35 completed questionnaires returned. These showed that 23% of CMTs had completed the MRCP exam series. Of these responses, 23 were from trainees who were in core medical training over the preceding 12 months, and included in our analysis. Nine per cent rated the relevance of formal CMTs' teaching to the PACES exam as 'good' or 'very good'. Seventeen per cent of trainees rated their exposure to ward-based PACES teaching as 'good' or 'very good', with 21% of trainees having attended more than four bedside-teaching sessions in the preceding 12 months, and 61% having attended no sessions at all. Furthermore, the quality of these sessions was reported as being 'good' or 'very good' by only 22% of trainees. Forty-eight per cent reported the available sessions as being of 'poor' quality.

Free text suggestions highlighted a lack of support structure in preparation for the PACES exam. Candidates reported a problem with finding suitable patients to practise examination skills in their respective hospitals. Some highlighted the need for formal history and communication skills practice sessions.

See online supplementary file which is an example of the questionnaire sent to trainees.

Design

Based on the initial data, key areas for change were identified and the following interventions were introduced over two Plan, Do, Study, Act (PDSA) cycles. Interventions for cycle 1 were introduced following the baseline measurements. The baseline questionnaire was redistributed and data analysed after 6 months. Cycle 2 interventions were then introduced and the same questionnaire was again used to assess for any improvement at 12 months.

In order to address the need for a general support network, a PACES mentoring scheme was implemented, whereby each trainee was allocated a mentor who had completed the MRCP series. Their role was to offer support and guidance in preparation for the exam in addition to providing one-to-one teaching opportunities on a more regular basis. Local volunteers were recruited (predominantly CMTs and junior registrars) allowing for each candidate to be allocated their own mentor.

This support framework was enhanced by the introduction of a confidential PACES forum on 'WhatsApp', which allowed both mentors and candidates to arrange sessions and share locations of useful practice cases without breaching any confidential patient information. To ensure this was the case, rules regarding sharing of patient information were strongly enforced and candidates were asked to agree to user conditions prior to joining the forum.

Furthermore, to offer candidates more support in practising history and communication skills with feedback, several out of hours, formal, classroom-based, group sessions were arranged. Candidates were offered the opportunity to sign up for these sessions and volunteers were recruited to be actors for the scenarios used.

Finally, consultants leading classroom-based curriculum-targeted CMT teaching were advised to use a focused, strategic teaching5 approach to PACES in the final half an hour of each teaching session.

Strategy

In PDSA cycle 1 the introduction of PACES mentors and the 'WhatsApp' group led to a marked improvement in trainee satisfaction and improved exposure to bedside-teaching. However, disappointing exam outcomes, and therefore pass rates, following the first exam sitting as well as trainee feedback identified the need for more senior-led bedside-teaching opportunities, to simulate PACES exam conditions.

This led to PDSA cycle 2 and the introduction of regular consultant-led bedside-teaching sessions via a weekly bedside teaching rota, to address the identified need for more senior-led teaching with performance feedback. All medical consultants in UHCW were approached to determine their availability and willingness to be involved in teaching. A core team of clinicians including consultants, STs and senior grade academic staff was allocated to an out of hours PACES teaching rota. Spaces were limited to a maximum of five trainees per session in order to optimise teaching delivery.

To assess overall effectiveness of our interventions, Likert Scale graphs were repeated for each aspect of study. Furthermore, the completion rate of MRCP by the time of final ARCP was compared with the previous year.

Results

The pass rates for MRCP PACES improved following our implementations. Of 22 CMTs 17 had passed PACES by their final ARCP date. This equated to a pass rate of 77.3%, which was a clear improvement compared with the baseline rate of 56.3% in the preintervention cohort, 12 months earlier.

These improvements were also reflected by the qualitative measurements of trainee satisfaction with exposure to and quality of bedside-teaching as well as relevance of curriculum-based teaching.

Ratings as ‘good’ or ‘very good’ for exposure to bedside-teaching improved from 17% to 46% after PDSA cycle 1 and to 59% after PDSA cycle 2.

There was an objective measure to verify this with rates of attendance to four of more PACES-oriented bedside-teaching sessions increasing from 21% at baseline to 46% after PDSA cycle 1 and to 55% after PDSA cycle 2.

Ratings as ‘good’ or ‘very good’ for quality of bedside-teaching sessions increased from a baseline of 21.7% to 61% after PDSA cycle 1 and 65% following PDSA cycle 2.

Overall ratings of relevance of curriculum-based teaching rated as ‘good’ or ‘very good’ showed an increase from 9% at baseline to 25% after PDSA cycle 1. This increased further to 34% following reinforcement of the intervention in PDSA cycle 2.

Graphs showing comparison of baseline to results at 6 months and 12 months following interventions are given in figure 1.

Conclusion

This project demonstrated a clear improvement in overall MRCP exam series pass rates by the end of CT2, bringing Coventry and Warwickshire (77.3%) in line with the national average (78%).

Furthermore, trainee feedback on exposure to, and satisfaction with, teaching demonstrated notable improvements following interventions.

Those who have delivered teaching or mentoring have also benefited from this project as they have had the opportunity to hone their leadership skills, keep up to date with knowledge and improve on teaching experience.

We have identified limitations in this quality improvement project and have recommended appropriate solutions. The most difficult limitation to overcome involves identifying which of multiple facets of improvement has been the most beneficial. Feedback from trainees in 'free text' suggests that having a peer-mentor to guide them through the stressful exam process, more senior support and regular bedside-teaching opportunities with performance feedback were most appreciated.

Although we identified that centralising the consultant-led bedside-teaching sessions potentially inhibits self-directed learning, it avoids consultants being overrun with individual requests and facilitates a workable rota for all to attend. This is particularly important given the limited availability of senior doctors owing to other commitments. As previously discussed, the use of a first-come-first-serve rota potentially limits less motivated trainees to fewer sessions. However, when used in conjunction with the peer-mentors through an interactive forum, these trainees can be identified and offered extra support.

Utilisation of a shared forum was key to the success of the framework, in that it enabled trainees to efficiently target their practice and avoid wasting precious revision opportunities between ward commitments. Its success though, relies on the collaborative input of all involved throughout. In our experience this is not always easy to maintain.

Existing literature supports the concept that focused strategic learning improves exam outcomes.5 The majority of CMT classroom-based teaching is blueprinted nationally, in order to ensure a broad foundation of core clinical knowledge. This however potentially limits its usefulness to PACES exam preparation. Negotiating with consultants delivering classroom sessions to include focused strategic learning during their sessions has allowed for the CMT curriculum to be intertwined with the PACES curriculum, thus ensuring dual relevance.

Improving local trainee satisfaction and MRCP PACES pass rates, as well as enhancing teaching opportunities, raises the profile of the trust as a whole. This can ultimately lead to improved recruitment and retention of trainees, while justifying local deanery funding for education.

We feel that our innovative and cost-effective interventions could be used on a wider scale to improve progression through core medical training and onto higher-level training. Furthermore, it encourages teamwork and leadership, both of which are essential skills required throughout a career in medicine. Ultimately our tailor-made teaching framework should not be used alone and should be used in conjunction with proactive self-directed learning, with significant preparation time.