The ERCP quality network benchmarking project: a preliminary comparison of practice in UK and USA
- 1Hepato-Pancreato-Biliary Unit, Freeman Hospital, Newcastle, UK
- 2Digestive Disease Center, Medical University of South Carolina (MUSC), Charleston, South Carolina, USA
- Correspondence to Dr Kofi W Oppong, Hepato-Pancreato-Biliary Unit, Level 6, Freeman Hospital, High Heaton, Newcastle upon Tyne, Newcastle, NE7 7DN, UK;
Contributors The ERCP QN was initiated by PBC. PBC and KWO designed the study. Statistical analysis was carried out by JR. KWO wrote the manuscript. All authors contributed to the drafting process and have seen and approved the final version.
- Received 14 December 2011
- Accepted 13 March 2012
- Published Online First 1 May 2012
Objective The Endoscopic retrograde cholangiopancreatography (ERCP) Quality Network is a voluntary system for submission of data to generate individual report cards and benchmarking. The aim of this study was to compare aspects of ERCP practice between USA and UK participants.
Design Analysis was limited to USA and UK based endoscopists who had each entered more than 30 cases. A number of practice and performance measures were studied including, rates of deep biliary cannulation, sedation use and success in bile duct stone removal.
Setting and patients Patients attending for routine and emergency ERCP in participating tertiary and secondary care units in the UK and USA.
Results 61 US endoscopists performed 18 182 procedures and 16 UK endoscopists 3172, respectively. The UK participants performed less complex procedures as judged by the accepted complexity grading system, 8% versus 35% at grade 3, p<0.001. There was a significantly greater use of sedation as opposed to anaesthesia in the UK 97% versus 34%, p<0.001. UK deep biliary cannulation rate was 93% versus 97%, p<0.001. For small bile duct stones (<10 mm) the UK success rate was 96% compared with 99%, p<0.001.
Conclusion The present data, while not purporting to be an accurate representation of practice in either country, documents good technical success in both cohorts, albeit significantly better in the USA. The inexorable drive to greater accountability and transparency of outcomes in endoscopic practice is likely to lead to increased participation in subsequent benchmarking projects.
There is increasing interest in documenting the performance of individual endoscopists, for their own education and to ensure quality standards.1 2 In the UK, the global rating scale3 has provided an assessment tool for endoscopy units helping to drive up the quality of the service and there is ongoing work under the auspices of the Joint Advisory Group (JAG) on Gastrointestinal Endoscopy to derive a framework for quality assurance of individual endoscopists. What has been lacking till now is an infrastructure to facilitate collection and analyses of data to allow individual endoscopists to easily compile a “report card” of their own practice or to benchmark themselves against their peers.
Several benchmarking projects are now in progress internationally. One is the Endoscopic retrograde cholangiopancreatography (ERCP) Quality Network (ERCPQN) which was initiated in 2007.4 Practitioners (or their staff) upload key data points on each case, to a central website, without identifying the patients. Data can be analysed to provide a confidential “report card”, or comparisons can be made with an average of all other contributors (benchmarking), without identifying them individually.
The aim of this study was to compare aspects of ERCP practice between US and UK participants.
Participation in the ERCPQN is voluntary and free of charge, and practitioners and patients are not identified. There is a secure website housed and supported by the Olympus Corporation of America. Interested endoscopists performing ERCP provide some initial basic demographic data (academic or community practice, years of practice, approximate volumes in training, lifetime practice and practice in the last year). They or their surrogates receive individual passwords and record structured data on each ERCP procedure. For this analysis we looked specifically at several different practice and performance measures, that is, the variation in case volumes and complexity, using an accepted 3 level system,5 outpatients versus inpatients, moderate sedation versus deep sedation or general anaesthesia, procedure and fluoroscopy times, rates of deep biliary cannulation (in untreated visualised papillas), minor papilla cannulation rates, success in removing small bile duct stones (<10 mm), and larger ones (>10 mm). Any case with prior papilla therapy (sphincterotomy or needle-knife) was excluded from the analysis of cannulation success; such cases were not excluded from other analysis. Classification of a procedure as successful required completion of the intended therapy. For example, placing a stent for stone disease was classified as a failure. Success was assessed separately for each procedure; repeat procedures were counted as separate events. Therefore, a patient with stones who underwent three procedures before the duct was cleared would count as a failure of therapy on the first two with success attributed to the final procedure. We chose not to report rates of adverse events, knowing the difficulty in ensuring accurate data for delayed events, which are the majority in ERCP.
Analysis is limited to USA and UK based endoscopists who have each entered more than 30 cases. The results were evaluated in aggregate.
Binomial CIs were calculated for proportions, and comparison between countries was performed by χ2 (or Fisher exact tests when appropriate) for proportions and Mann–Whitney tests for continuous variables (using SPSS software). Bonferroni correction for multiple testing was used.
By May 2011, a total of 151 endoscopists had registered; this analysis is limited to the 61 endoscopists based in USA and the 16 in UK who have each entered more than 30 cases (total cases 18 182 and 3172, respectively). Both the US and UK endoscopists averaged 10 cases/month, but the range in USA was much greater (1–67 vs 6–16). Sixty-six doctors reported their annual volume. Nineteen of this group performed <100 ERCP a year, 12 (63.1%) had a cannulation rate ≥90%. Of the 47 doctors performing more than 100 procedures a year, 85.1% had a cannulation rate ≥90% p=0.09. None in the UK versus 12 (20%) in USA were doing less than 75 per year. There was a greater proportion of practitioners in tertiary practice in the US cohort 26/61 (42.6%) versus 5/16 (31.3%) in the UK; however, this was not statistically significant p=0.6.
Table 1 compares some aspects of practice in the UK and USA. The UK participants were performing less complex procedures as judged by the accepted complexity grading system, (8% vs 35% at grade 3, p<0.001). A significantly higher proportion of grade 3 cases were for Sphincter of Oddi manometry in USA (44%) compared with the UK (22%) p<0.001. Propofol/general anaesthesia was used significantly less in the UK (3% vs 66%, p<0.001) and a significantly higher proportion of UK procedures were on inpatients 54% versus 35% (p< 0.001). In the UK, procedures performed under propofol or general anaesthesia were slightly less successful but not significantly so (possibly reflecting the selective use for more challenging cases). Procedure duration and fluoroscopy time were also significantly shorter (p<0.001) in the US cohort. Comparative success rates for key technical metrics are shown in table 2. UK endoscopists recorded a lower average deep biliary cannulation rate (93% vs 97%, p<0.001), and success in removing bile duct stones, particularly those >10 mm in diameter (86% vs 97%, p <0.001). Since contributors usually only recorded the success for removing stones in cases of successful cannulation, the overall success rates were substantially lower. There were remarkably few reports of attempts to cannulate the minor papilla in UK. The wide variation in reported individual biliary cannulation rates for USA and UK are shown in figure 1.
Internationally there is increasing interest in quality assurance and improvement in endoscopy and there are growing calls for quality to be documented with data. Audits mainly focusing on complications have been published by academic centres, practices and research groups in many countries6,–,10 Despite recommendation to do so2 11 many endoscopists have been resistant to the idea of generating individual report cards for fear that they might become public and be used against them. Public mood, regulatory bodies and the tide of professional opinion across all medical specialities is inexorably moving to greater accountability and publication with transparency of outcomes.
ERCP is the most complex and risky procedure performed regularly by endoscopists. It is very important that procedures are done successfully and safely and that adverse events are minimised. Previous commentators have suggested that the quality of some aspects of ERCP practice in UK are suboptimal,12 and there are similar concerns in USA.13 The ERCPQN system was introduced initially in USA, and later made available to UK endoscopists. Twenty-six registered, but only 16 have so far entered data on at least 30 procedures. We felt it would be of interest to compare the results of the US and UK endoscopists, bearing in mind that these are by no means, representative samples.
The results document a significantly higher biliary cannulation success rate for US participants as compared with UK and higher technical success as measured by stone extraction. The UK data represent acceptable levels of performance, and compare favourably with the prior audit in which only 42% of endoscopists achieved a cannulation rate of >90%.6
A major limitation of the current version of the ERCPQN is lack of any data validation. The overall high performance reported may reflect self selection by individuals comfortable enough with their data to enter it, but there remains the possibility of selective entry or even intentional errors. It is perhaps reassuring from that standpoint that some endoscopists did report rather low biliary cannulation rates (figure 1). The anonymity of the data and of comparisons should have mitigated against errors of data entry, and self-reporting has been used successfully in colonoscopy outcomes studies.14 15 A separate multivariate analysis of this data-set does suggest that annual volume and experience may predict biliary cannulation success.16 Because of the difficulty in capturing delayed adverse events such as pancreatitis, we chose not to report the data.
While we concede that these data do not purport to be an accurate cross-section of national practice in either country, we did find two striking ‘cultural’ differences in practice. US endoscopists reported a much higher proportion of complex ‘grade 3’ cases (mainly pancreatic therapy and sphincter dysfunction), whereas the main workload in UK was standard biliary interventions, for example, for stones and jaundice. The use of anaesthesia services, that is, 3% versus 66%, is another marked difference. This is an area of USA ERCP practice that many UK endoscopists envy. Deep (or general) sedation using propofol (called monitored anaesthesia care) in many US units is administered by specially trained nurses (certified registered nurse anaesthetists), under the nearby supervision of an anaesthetist. In the UK, propofol is usually administered by a physician anaesthetist, and the difficulties and cost inherent in such an arrangement have severely limited its use. A recent working party of the British Society of Gastroenterology and Royal College of Anaesthetists has produced guidelines on the use of propofol17 which will hopefully form the basis for increased access to deep sedation in the UK. In turn, this should facilitate progress towards approaching more complex ERCP cases which are easier to perform and probably safer when done under anaesthesia.
What is already known
▶ ERCP is the most risky routinely performed endoscopic procedure.
▶ Quality assurance in Endoscopy is a topical subject with calls for better documentation of performance, greater accountability and transparency of outcomes.
What this study adds
▶ This study establishes that the ERCP-QN is a suitable system for collating some of the key performance metrics required for ERCP quality assurance.
▶ The study documents on average good quality outcomes in both cohorts whilst demonstrating differences in outcome and practice between the UK and USA participants.
How this might impact on clinical practice in the foreseeable future
The wider adoption of the ERCP-QN and other benchmarking projects should help drive improvements in ERCP quality.
While the current ERCPQN is strictly confidential to the contributing endoscopists, such quality data are of increasing interest to providers and professional organisations and to patients and their advocates. This is exemplified by the recent online publication in USA by a consumer group of the results of coronary-artery bypass grafting at 221 US cardiac surgery programmes.18 This voluntary reporting of risk-adjusted outcomes in approximately 20% of US cardiac surgery programmes has been described as a watershed event in healthcare accountability.19 Where cardiac surgery has led, other interventional practices including endoscopy are likely to follow. In USA, the American Society of Gastrointestinal Endoscopy (ASGE) and American College of Gastroenterology (ACG) have initiated a joint national system for benchmarking called GI quality improvement consortium (GIQuIC).20 This opened for enrolment in July 2010, and focuses initially on colonoscopy. ERCP will be incorporated along with other procedures eventually, building on the experience in ERCP QN.
In the UK, a powerful driver for collection of quality data will be revalidation. It is envisaged that there will be a 5-year revalidation cycle for endoscopists. The process is currently under development by JAG with each procedure having specific key performance indicators (KPIs). The proposed KPI for ERCP21 specifies a deep cannulation rate of >90%, performance of more than 75 procedures a year and a pancreatitis rate of <5%. It is very likely that collection of the type of data captured by the ERCP QN will become the de facto standard in the near future. The data will be easier to collect (and perhaps more believable) when it can be extracted directly from the endoscopy reporting system. This has already been established for some systems in USA, feeding data to GIQuIC. The capture of late adverse events will require additional mechanisms, and likely some form of audit. The ERCP Quality Network project has shown that voluntary quality benchmarking can be achieved, and provides a signpost for the broader schemes that will be needed in the future.
The authors wish to thank all the participants in the ERCPQN.
Funding This study was supported by the Olympus Corporation of America.
Competing interests PBC and JR have undertaken consultancy work for the Olympus Corporation of America.
Ethics approval The study was approved by the institutional review board of the Medical University of South Carolina.
Provenance and peer review Not commissioned; externally peer reviewed.