Abstract
Background Shared decision making is an important aspect of patient centredness. Lack of this consulting behaviour is a common reason for failure in the Membership of the Royal College of General Practitioners (MRCGP) consulting skills examination.
Aim To investigate candidates' performance in shared decision making and overall performance in the MRCGP consulting skills assessment compared with an independently validated measure, the OPTION (‘observing patient involvement’) scale.
Design Cross-sectional study.
Setting MRCGP examination, UK.
Participants Two hundred and fifty-two consultations submitted by 36 GPs submitting seven consultations per videotape.
Method A stratified sample of 63 candidates, 21 each from fail, pass and merit selected from candidates in the MRCGP consulting skills examination, were approached for participation. Participants' examination videotapes were independently assessed for shared decision making using the OPTION scale by two non-clinical raters.
Results Thirty-six candidates (of 63; 57%) who participated were no different from non-participants. Candidates who passed the ‘sharing management options’ in the MRCGP had significantly higher OPTION scores than those who did not (35.4 versus 27.3; mean difference = 8.1, P = 0.044). There was a significant difference between OPTION scores of MRCGP candidates with ‘fail’ and ‘pass’ (including pass with merit): 28.6 versus 36.1, 95% confidence interval CI = 1.13 to 13.87. Scores decreased as clinician age increased but were not significantly associated with sex of GP, age or sex of patient or consultation duration. The probability of passing the MRGCP increased as OPTION scores increased.
Conclusion This study demonstrated concurrent validity of the MRCGP consulting skills assessment of sharing management options against an independent validated instrument for shared decision making, the OPTION scale. Candidates who performed best in the MRCGP exhibited high scores with OPTION. This study provides the basis for further work to demonstrate evidence for the potential of training for professional assessment to improve consulting competence.
- assessment of competence
- cross-sectional studies communication
- referral and consultation
- shared decision making
INTRODUCTION
One of the most common reasons for failing the consulting skills assessment of the Membership of the Royal College of General Practitioners (MRCGP) examination is an inability to fulfil or demonstrate the shared decision-making component. It is therefore important to investigate how this area is being assessed in the examination and to examine the characteristics of candidates who fail, pass or do exceptionally well.
Involving the patient in the management plan to the appropriate extent, operationalised as ‘the doctor shares management options with the patient’ is one of a number of competencies tested in the consulting skills module of the MRCGP examination.1 This is a summative assessment taken by the majority of GP registrars in the UK towards the end of their training and also by registered GPs from this country and abroad. Communication skills training in UK general practice is geared to improving patient centredness although this is often not achieved by examination candidates.2
‘Shared decision making’ has been regarded as an element of ‘patient centredness’ in the general practice consultation.3,4 It is promoted because it enhances patient autonomy, is associated with more positive consultations from a patient perspective,5 and when used appropriately is welcomed by patients6,7 without associated increase in anxiety.8 Patients vary in the extent to which they wish to be involved in decision making and although they value this less than other attributes of the consultation such as being listened to, and being given understandable information they place greater value on it once they have experienced it.9 It is closely related to concordance10 but has yet to show an impact on health outcomes.11
How this fits in
Shared decision making is a key aspect of patient-centred consulting behaviour. There has been little previous research to validate this aspect of consulting skills assessment within a professional examination. This study demonstrated the concurrent validity of the MRCGP consulting skills assessment of ‘sharing management options’ against an independent validated instrument for shared decision making, the OPTION scale.
Professional medical examinations, such as the MRCGP consulting skills module, assess consultation skills competences and have included criteria that claim to assess ‘shared decision making.’ However, there has been little research done to independently validate this aspect of such assessments.12,13 This study arose from a recognised need to evaluate the assessment of shared decision making in this postgraduate exam.
The study obtained access to a sample of examination candidates' videotape assessments and provided an opportunity to compare the results obtained by candidates in the MRCGP against scores obtained by the use of the ‘observing patient involvement’ (OPTION) scale. The scale is a validated instrument that has been developed for measuring the extent to which clinicians share decision making with patients in primary care consultations.14,15
Did OPTION scores predict success or failure in the MRCGP examination? What can be learnt about the examination process by using such a tool and are there any patient or practitioner variables that influence the results? This study aimed to determine whether concurrent validity of assessment of patient involvement in the MRCGP consulting skills exam could be established using the OPTION scale, a validated instrument.16
METHOD
This was a cross-sectional study comparing candidate performance in MRCGP (under the assessment conditions at the time) with scores generated using the OPTION scale for the same set of consultations.
For the consultation skills component, candidates submit a videotape consisting of seven consultations. These seven consultations are those interactions that they consider best represent their ability to demonstrate 15 performance criteria, one of which is ‘sharing management options,’ that is considered to be an important part of the construct of shared decision making. The consultations include at least one mental health and one paediatric consultation and are not assessed on the level of challenge. There are also three merit areas, namely when the candidate ‘explores the patient's health understanding’, ‘takes account of some or all of the patient's elicited beliefs' and seeks to confirm the patient's understanding’ (Supplementary Box 1).
In the examination, the ‘sharing management options’ criterion is marked as present or absent in each of the seven consultations by seven independent clinical raters who are all GPs specifically trained to undertake this assessment. The other criteria are similarly marked as present or absent. The overall score is based on the assessments of the seven raters for the 15 performance criteria in seven consultations for each candidate. Each candidate needs to demonstrate each criterion in a specified number of consultations (a threshold level). The threshold is usually presence of the attribute in four out of seven consultations but may be less. For example, presence in two or three out of seven consultations may be judged as a pass in more ‘difficult’ criteria such as the candidate ‘detects cues’, assesses ‘psychological and social’ context and ‘shares management options’. If candidates demonstrate the three separate ‘merit’ criteria on a minimum number of instances (usually nine) they are awarded a pass with merit. Candidates are therefore awarded pass, fail and pass with merit on the basis of this assessment (Table 1).
The sample consisted of selected candidates from the MRCGP examination in October 2003: all 780 candidates were categorised by their result, 473 (60.6%) pass, 154 (19.7%) fail and 153 (19.6%) merit. From these three categories, we hoped to recruit at least seven candidates from each category and therefore 63 candidates (three times as many) were randomly selected (using the random sampling tool in Excel) and were approached to consent.
Two non-clinical raters, blind to the MRCGP assessment, assessed each of the seven consultations per candidate using the OPTION scale. The raters had undergone specific training and were experienced in this assessment. This is a 12-item instrument, with items derived from qualitative studies of GPs in training17 and medical educators.18 It has been shown to have good inter-rater agreement, high internal consistency and construct validity14 and was therefore considered valid for use as an independent measure of shared decision making. For OPTION ratings, 12 items are given scores for the magnitude of a skill exhibited (between 0 to 4) by two raters. The mean score of the two raters is converted to an overall percentage and is taken to be the overall OPTION score for the consultation (Box 1). All seven consultations per candidate were assessed using both methods independently (Figure 1).
Box 1. OPTION INSTRUMENT items.14
▸ The clinician identifies a problem(s) needing a decision-making process.
▸ The clinician states that there is more than one way to deal with an identified problem (‘equipoise’).
▸ The clinician lists ‘options’, including the choice of ‘no action’ if feasible.
▸ The clinician explains the pros and cons of options to the patient (taking ‘no action’ is an option).
▸ The clinician checks the patient's preferred information format (words/numbers/visual display).
▸ The clinician explores the patient's expectations (or ideas) about how the problem(s) are to be managed.
▸ The clinician explores the patient's concerns (fears) about how problem(s) are to be managed.
▸ The clinician checks that the patient has understood the information.
▸ The clinician provides opportunities for the patient to ask questions.
▸ The clinician asks for the patient's preferred level of involvement in decision making.
▸ An opportunity for deferring a decision is provided.
▸ Arrangements are made to review the decision (or the deferment).
The following variables were derived from the MRCGP data: the average score for the ‘shares management options’ criterion, the number of consultations out of seven in which this criterion was present, and the candidate's result (pass with merit, pass, or fail). Data were analysed using ANOVA for association between fail, pass and merit categories with mean OPTION scores.
To examine whether the total OPTION score of clinicians was influenced by patient age or sex and clinician age or sex, a multilevel model approach with patients nested within clinicians was carried out using MLwiN (from the Centre for Multilevel Modelling, Institute of Education, University of London). A regression model for predicting total score was fitted with explanatory variables including patient age, patient sex, GP sex, and GP age, using patient data for level 1 and clinician data for level 2. Logistic regression analysis was undertaken to predict the probability of passing MRCGP on the basis of OPTION scores achieved.
RESULTS
Thirty-six of the 63 (57%) MRCGP candidates approached consented for their videotape to be used for this study, giving a total of 252 consultations for analysis. These were 15 candidates from the ‘pass’ category, 10 who failed and 11 who achieved merit in the examination (see Figure 1 for details). These participants were similar to those who did not consent in terms of their age, sex and performance in the MRCGP consulting skills examination (Table 2).
Primary outcome
Candidates who passed the ‘shares management options’ performance criterion (that is; demonstrated this criterion in at least two out of seven consultations) had significantly higher OPTION scores than those who did not (OPTION scores 35.4 (n = 30) versus 27.3 (n = 6); mean difference 8.1, P = 0.044).
Further analyses
There was a strong positive correlation between the numbers of consultations (out of seven) in which candidates demonstrated ‘shares management options’ (MRCGP criteria met) with the OPTION score (rank correlation [Spearman's ρ] = 0.67, P [two-tailed] <0.001) (Supplementary Figure 1).
There was a significant difference between ‘fail’ and ‘pass including pass with merit’ OPTION scores, 28.6 vs. 36.1; 95% confidence interval (CI) 1.13 to 13.87, P = 0.022. The mean OPTION scores for each of the three result categories of MRCGP (namely, fail, pass and merit) were 28.6, 32.8 and 40.6, respectively. This indicated statistically significant differences between ‘fail’ and ‘merit’ (95% confidence interval = 3.39 to 20.7; P = 0.004) but only a borderline difference between ‘pass’ and ‘merit’ score (95% CI = −4.0 to 15.6; P = 0.05) (Table 3).
There was also a correlation between candidates who achieved higher numbers of merit performance criteria (out of 21) and their OPTION score (Pearson correlation = 0.49, P [two-tailed] = 0.002). Merit criteria are considered to be indicators of patient centredness and a correlation between other such indicators was to be expected.
Multilevel analysis, using variables patient age, patient sex, clinician age, clinician sex and duration of consultation, revealed that the age of the clinician was the only significant variable; as the age of the clinician increased, the OPTION score decreased (coefficient estimate = −0.65, standard error = 0.2, P-value = 0.0049) (Table 4). The probability of passing the MRGCP increased as OPTION scores increased. For an OPTION score of 36, the probability of passing MRGCP was estimated to be 0.9 increasing to 0.95 if the score was 42 (Supplementary Figure 2).
DISCUSSION
Summary of main findings
This study of MRCGP candidates, randomly approached, showed that doctors who passed ‘sharing of management options’ in the MRCGP consulting skills assessment also exhibited more evidence of sharing decisions with their patients, as assessed by the OPTION scale. Global performance in the MRCGP consulting skills examination was also associated with better performance in the OPTION scale, suggesting that shared decision making may indicate the existence of a more generalised level of global competence.
Candidates with higher OPTION scores were more likely to pass the MRCGP and candidates scoring MRCGP merit had the highest scores on OPTION, whereas increasing age of MRCGP candidates was associated with lower OPTION scores. It appears that achieving a high OPTION score could predict a pass in the MRCGP.
Strengths and limitations of the study
The strengths of this study included the development of a clearly stated research aim, the use of data taken from a professional validated examination process, a careful sampling process and the use of an instrument that has been previously used to investigate shared decision making (the OPTION scale). The limited size of the sample in each of the examination result categories is a potential weakness.
Comparison with existing literature
There are few published evaluations of professional examinations using other measures in parallel. McKinstry and co-workers demonstrated a lack of correlation between MRCGP assessment and patient satisfaction using a patient enablement index and consultation satisfaction questionnaire.19 A post-trial evaluation revealed that GPs perceived that there is limited patient demand and scope for this behaviour in practice,20 but it is known from surveys that patients have increasing expectations that they will be involved in decision making.21
Some patients are more likely to want an opportunity to share decision making. In an interview study, younger patient age, social class and smoking status were related to an expressed preference for sharing.22 OPTION does specifically address the issue of patient preference. Focused skills training enhances doctors' confidence and ability23 to involve patients but it is not known whether this is sustained in routine practice and the impact on long-term patient outcomes requires further work.24
Some authors consider that patient ‘activation’ as well as ‘involvement’ may be necessary to produce improved outcomes25 and other factors, such as trust, may play an important part in patients' willingness to share decision making.26 It is also possible training that has the wider aim of improving patient centredness in a more global sense may be a more effective strategy to improve patient-based outcomes 27
Implications for assessment, practice and future research
In terms of policy, the results provide reassurance that the examination process is testing candidates in the area of shared decision making. Other areas of the consulting skills assessment may require similar validation.
Educationally, the use of the OPTION rating scale might be useful for those preparing for the examination to self assess and improve their skill at involving patients. The OPTION tool provides a reductionist approach to assessment of patient involvement compared to the MRCGP but by defining in more detail what is meant by ‘involvement’ it also covers a broader notion of this concept.28 OPTION could also be used in education and training of examiners, trainers and candidates for assessing and improving patient involvement.
These results support further research into whether training in patient involvement will improve examination results.
This study demonstrated concurrent validity of the MRCGP consulting skills assessment of shared decision making against an independent validated instrument, the OPTION scale. A focus on shared decision making as an element of patient centredness may help candidates to prepare for this professional assessment. This study provides the basis for further work to demonstrate evidence for the potential of training for professional examinations to improve consulting competence.
Acknowledgments
We are grateful to Alan Sykes (former Senior Lecturer in Statistics at Swansea University) who undertook the data analysis and multilevel modelling for this study. We would like to thank the examination department and examination board of the RCGP who supported the study and provided anonymised data on participant scores; also the raters who undertook the OPTION analysis of videotapes. The study was presented at the International Ottawa Conference on Clinical Competence, New York 20–24 May 2006.
Notes
Supplementary information
Additional information accompanies this article at http://www.rcgp.org.uk/bjgp-supp-info. The criteria for the MRCGP consulting skills examination have been revised since this study was conducted. For current criteria please refer to http://www.rcgp.org.uk/exams/
Funding body
The University of Hull, University of Wales (Cardiff and Swansea) and Trent Research and Development Support Unit
Ethics committee
Eastern Multicentre Research Ethics Committee (study number 03/5/079). The study was also approved by the RCGP examination board and for research management and governance by West Lincolnshire PCT
Competing interests
Niroshan Siriwardena and Adrian Freeman are members of the panel of examiners for the MRCGP. Adrian Freeman is convenor and Peter Campion was immediate past convenor of the MRCGP consulting skills examination. Glyn Elwyn and Adrian Edwards developed and evaluated the OPTION scale for assessment of shared decision making and this is available on a not-for-profit basis via Cardiff University29
- Received August 5, 2005.
- Revision received August 17, 2005.
- Accepted March 13, 2006.
- © British Journal of General Practice, 2006.