Article Text
PDF
Abstract
Objective The aim of this study was to determine the number of OGDs (oesophago-gastro-duodenoscopies) trainees need to perform to acquire competency in terms of successful unassisted completion to the second part of the duodenum 95% of the time.
Design OGD data were retrieved from the trainee e-portfolio developed by the Joint Advisory Group on GI Endoscopy (JAG) in the UK. All trainees were included unless they were known to have a baseline experience of >20 procedures or had submitted data for <20 procedures. The primary outcome measure was OGD completion, defined as passage of the endoscope to the second part of the duodenum without physical assistance. The number of OGDs required to achieve a 95% completion rate was calculated by the moving average method and learning curve cumulative summation (LC-Cusum) analysis. To determine which factors were independently associated with OGD completion, a mixed effects logistic regression model was constructed with OGD completion as the outcome variable.
Results Data were analysed for 1255 trainees over 288 centres, representing 243 555 OGDs. By moving average method, trainees attained a 95% completion rate at 187 procedures. By LC-Cusum analysis, after 200 procedures, >90% trainees had attained a 95% completion rate. Total number of OGDs performed, trainee age and experience in lower GI endoscopy were factors independently associated with OGD completion.
Conclusions There are limited published data on the OGD learning curve. This is the largest study to date analysing the learning curve for competency acquisition. The JAG competency requirement for 200 procedures appears appropriate.
- ENDOSCOPY
Statistics from Altmetric.com
Significance of this study
What is already known on this subject?
The oesophago-gastro-duodenoscopy (OGD) completion rate of 95% is a key indicator of competency and an accepted quality indicator.
The number of OGDs required to achieve competency is not well established and there is a paucity of published data on OGD learning curves.
OGD training programmes around the world use the total number of OGDs performed as part of competency criteria. The number required in the UK is 200.
What are the new findings?
Our study interrogated the Joint Advisory Group (JAG) e-portfolio, which includes data from all training centres in the UK. This provided a unique opportunity to analyse OGD data submitted by 1255 trainees in the early stages of their training.
Two statistical methods were used to analyse the OGD learning curve. By moving average method, trainees attained an average 95% completion rate at 187 procedures. By learning curve cumulative summation (LC-Cusum) analysis, after 200 procedures, >90% trainees had attained a 95% completion rate.
OGD completion was associated with the total number of OGDs performed, trainee age and with lower GI endoscopy experience. In terms of patient factors, female gender, younger age and absence of sedation were associated with completion.
How might it impact on clinical practice in the foreseeable future?
Our study lends statistical support to the current UK and Australian requirement for 200 procedures.
Defining the OGD learning curve allows trainees to benchmark their progress against a UK national average.
LC-Cusum charts provide real-time feedback on individual learning curves. This information will be useful to trainees, trainers and training bodies.
Introduction
It is incumbent on endoscopy training programmes to assess the competency of trainees. Markers of technical competency in upper GI endoscopy (oesophago-gastro-duodenoscopy, OGD) include completion of procedures without physical assistance, successful J-manoeuvres and satisfactory scores in direct observation of procedural skills (DOPS) assessments.
In the UK, the Joint Advisory Group on GI endoscopy (JAG) require trainees to intubate the second part of the duodenum (D2) and perform a J-manoeuvre at least 95% of the time.1 It is accepted that not all OGDs can be completed to D2 due to patient intolerance, stricture or previous gastric surgery.2 ,3 Nevertheless, successful completion of OGDs is a prerequisite for competency. In the USA, OGD completion is one of two quality measures captured by the Clinical Outcomes Research Initiative. The US Task Force on Quality in Endoscopy has recently published a target OGD completion rate (OCR) of 98%.4 Specifying a target completion rate is necessary but insufficient in the training context without stipulating the number of procedures over which the target rate is to be achieved. This number is not well established. JAG currently requires a minimum of 200 procedures submitted through the JAG endoscopy training system (JETS). One hundred and thirty procedures are required in the USA,5 150 in Canada6 and 200 complete unassisted procedures in Australia.7
An early study of OGD learning curves showed that completion rates only approached 90% after 100 procedures and 95% after 120 procedures.8 In a larger study, 135 first-year GI fellows from different centres were followed-up to determine when competency was attained. Competency was defined as completion to the duodenum and correct recognition of abnormalities. By this definition, 160 procedures were required to attain a competent performance of >90%.9
JETS is an e-portfolio used by all trainees and endoscopy training centres in the UK.10 For the purpose of this study, the term trainee was defined as a healthcare professional, who at the time of commencement of data submission to the JETS database was performing endoscopy under supervision. It is incumbent upon trainees to submit data from all endoscopic procedures they have performed, allowing trainees to demonstrate their experience and competency progression. We have previously used the JETS database to analyse the learning curve to achieve competency in colonoscopy.11 JETS was rolled out in 2009 and currently contains OGD procedure data for >2300 trainees. The aim of this study was to analyse the learning curve to achieve competency in OGD completion using all trainee data submitted to JETS. Two statistical approaches were used – the moving average method and the learning curve cumulative sum (LC-Cusum) method as previously described.11 ,12
Methods
Subjects
The JETS e-portfolio was interrogated to retrieve all OGD records from 1 September 2009 to 7 May 2014 submitted by trainees. Users of the JETS e-portfolio agree to a privacy statement declaring that trainee data may be used for audit and research purposes. Ethics committee approval was not required for this research as JETS e-portfolio users permit their data being used for such purposes and data used for analysis was anonymised and contained no patient identifiable data. The following fields were retrieved: endoscopist identifier, date of procedure, patient age and gender, use of sedation, diagnosis at OGD, extent of procedure, extent completed by trainee and J-manoeuvre performed by trainee. For each trainee, the number of different trainers, completion of a JAG Basic Endoscopy course, age, lower GI endoscopy experience and baseline number of procedures were retrieved. Some trainees had entered the number of OGDs performed prior to submission of data to the e-portfolio (baseline number) although this was not a mandatory requirement. All trainees were surveyed to determine their baseline number of OGDs. Two reminders were sent.
Trainees who had a known baseline number of >20 procedures were excluded from subsequent analyses to assess only those trainees who were in the early stages of their OGD training. Trainees who had submitted a total of fewer than 20 OGDs were also excluded from further analysis (the minimum number required to calculate a moving average completion rate).
We wanted to ensure that learning curves were assessed only for trainees in the early stage of their training, that is, with a baseline experience of ≤20 procedures. Omitting trainees with an unknown baseline from this analysis may well omit trainees with a baseline experience of ≤20 procedures. Given that a significant proportion of trainees had an unknown baseline experience, competency analysis was performed including and excluding these trainees. Differences in various measured variables between trainees with known and unknown baseline were tested for statistical significance.
Primary outcome measure
In accordance with the definition of a complete procedure provided by JAG,1 the primary outcome measure was OGD completion, defined as passage of the endoscope by a trainee (trainee extent) to D2 without physical assistance. It is mandatory for trainers to observe procedures performed by a trainee. If D2 was reached by a trainee but physical assistance from a trainer was given at any stage of the procedure, the OGD was defined as incomplete. The procedure outcome (complete/incomplete) was submitted to the e-portfolio by the trainee. The OCR was defined as the proportion of OGDs successfully completed according to the above definition of completion. A secondary outcome measure was completion of a J-manoeuvre by the trainee.
Statistical analysis
Moving average and LC-Cusum methods were used to analyse the trainee competency learning curve. The proportion of trainees defined as competent in terms of an OCR of ≥95% was determined by both methods.
The mean OCR was calculated for each trainee over blocks of 20 procedures to conduct a moving average analysis. Blocks of 20 procedures were the smallest possible to allow for a completion rate equal to the JAG standard of 95% (=19 of 20 completed procedures). The mean OCR for all trainees was plotted against the number of procedures performed to determine the procedure number after which the mean OCR reached ≥95%.
LC-Cusum score calculation and its application to endoscopy skills acquisition has been described previously.11 ,13 Unlike other Cusum techniques, the LC-Cusum is specifically designed for the training environment by considering each procedure in time sequence and not penalising the trainee for early unsuccessful procedures by having a holding barrier at zero. When the LC-Cusum crosses a predefined threshold, it is concluded that there is sufficient evidence that the procedure has been learned. LC-Cusum scores were plotted for each trainee and a trainee was defined as competent if the graph crossed the threshold limit. The number of procedures performed at the point at which the graph crosses this threshold is the competency run limit (crl). Threshold values are determined by four parameters: p0 (the acceptable failure rate), p1 (the unacceptable failure rate), α (type I error) and β (type II error). These parameters were set at 0.05, 0.10, 0.10 and 0.10, respectively, for this study. p0 was set to 0.05 as this is the performance standard set by JAG. p1 is usually set to twice the value of p014 ,15 and appears appropriate from our previous work on colonoscopy learning curves.11 Typical values for type I and type II errors are 10% and, unlike power calculations for clinical trials, in the context of performance monitoring they are typically set to be equal.16 The LC-Cusum was designed in a similar manner to the two one-sided test procedure to test for equivalence in clinical trials.17
The moving average method was also applied to the learning curve of successful J-manoeuvres performed by the trainee.
Anatomical extent and factors associated with OGD completion
The extent of the procedure performed by the trainee is recorded in all cases. The extent achieved by trainees in cases where the trainee failed to intubate D2 without physical assistance was tabulated to determine at which anatomical site trainees have most difficulty in progressing with the procedure.
To determine which factors are independently associated with OGD completion in the learning context, a mixed-effects logistic regression model was constructed with OGD completion as the outcome variable. Trainees and training centres were entered into the model as random effect terms. The following were entered as fixed effects: number of procedures, training intensity (number of procedures per month), trainee specialty (physician, surgeon, nurse endoscopist), trainee age, breaks from training of >6 months, experience of ≥50 flexible sigmoidoscopy or colonoscopy procedures recorded in the JETS e-portfolio, completion of a JAG-accredited basic endoscopy course, patient age, patient gender, use of sedation (intravenous benzodiazepines, opiates or propofol) and endoscopic diagnosis.
Forty-six different diagnoses could be entered into the JETS diagnosis field and multiple diagnoses were permissible. We collapsed the diagnosis field into nine different binary (present/absent) covariate fields: achalasia, Barrett’s oesophagus, diverticulum, extrinsic compression, malignancy, peptic ulcer disease, polyps, varices and other diagnoses. Previous publications have classified OGD diagnoses into 1818 or 21 categories.19 We chose nine categories as these could be grouped easily from the diagnosis field options in the JETS database and the overall number of categories was minimised to simplify the multivariate analysis.
Training intensity is highly dependent on breaks from training, such as rotating into a training post where there is no opportunity to perform endoscopy. Therefore, an adjusted training intensity was calculated by dividing the number of procedures performed by the time period of continuous training. Continuous training was defined as the time from start to end date of submitted data, minus any periods of 90 days or more during which no data were submitted. It was assumed that any breaks from training of <90 days were unlikely to be due to a change in training post.
Some of the factors entered into the mixed effects model were categorised to allow more meaningful interpretation of the calculated ORs. The number of procedures was categorised into blocks of 50 procedures. Similarly, the adjusted training intensity was categorised into blocks of 10 procedures per month. The age of the trainee was categorised into tertiles (<30, 30–35 and >35 years).
A p-value <0.05 was considered to be statistical significant. All p-values and ORs are quoted from the mixed-effects model. All data computations, graphics and statistical analysis were performed using R (R foundation for Statistical Computing, Austria), and the mixed modelling used the lme4 package.20
Validation of data submitted to the JETS database
To assess, at least partially, whether the JETS database captured all OGDs performed by trainees, validation of the data submitted to JETS by a sample of trainees was performed. We were able to obtain OGD data for all trainees from the local endoscopy reporting database at the Queen Elizabeth Hospital Birmingham over the most recent 4 years. Of these trainees identified from the local database, some trainees also submitted data to JETS. The number of OGDs submitted to JETS was compared with the number of OGDs logged for the trainee in the local endoscopy reporting database over the same time period. We compared the number of procedures in the two data sources using a linear scatterplot and Pearson's correlation coefficient and a mean difference Bland–Altman plot.21
Results
The e-portfolio contained 493 374 OGD records performed by 2336 trainees, representing 311 different training centres. The number of OGD records, trainees and training centres is shown in the study flow diagram (see figure 1). Baseline data had been recorded in the JETS e-portfolio by 710 trainees (30.4%). Nine hundred and fifty-four trainees responded to the survey (40.8%). There was significant overlap in those trainees who had recorded baseline data and those who responded to the survey so that the overall baseline data were available for 1131 trainees (48.0%).
Study flow diagram.
Given that a significant proportion of trainees had an unknown baseline experience, competency analysis was performed including and excluding these trainees. There were 1255 trainees with a baseline experience of 20 or fewer procedures, or with an unknown baseline experience, who had performed ≥20 OGDs. This group was used for subsequent analyses of competency. There were 328 trainees with a known baseline experience of 20 or fewer procedures who had also performed ≥20 procedures, and this group also underwent analysis of competency. Trainees with known baseline experience differed significantly from trainees with unknown experience for several factors, as shown in online supplementary table S1. Trainees with unknown baseline comprised more nurse endoscopists and were older but had performed fewer overall OGDs with less lower GI endoscopy experience.
Supplementary table
Analysis of competency
By moving average analysis, when trainees with an unknown baseline were included, the 95% OCR was achieved after an average of 187 procedures. With exclusion of trainees with an unknown baseline, the 95% OCR was achieved after 186 procedures (see figure 2).
Mean moving average OGD (oesophago-gastro-duodenoscopy) completion rate (OCR) against number of OGDs performed (black line), plotted with 1 and 2 SDs of the mean. The OCR reached 95% at 187 procedures when trainees with unknown baseline were included (A) and at 186 procedures when trainees with unknown baseline were excluded (B). The numbers in italics above the x-axis are the denominators of the number of trainees over which the OCR is averaged.
The crl is the number of procedures at which the LC-Cusum graph crosses the upper control limit, defining the trainee as competent. The median crl was 131 when trainees with unknown baseline were included and 149 when excluded (see figure 3). It is important to realise that a crl can only be calculated for those trainees who attained competence. The proportion of trainees who had attained competency, as defined by LC-Cusum, was therefore calculated after a given number of procedures (see table 1). After 200 procedures, >90% trainees had attained LC-Cusum defined competency, whether or not trainees with unknown baseline were excluded. Moving average and LC-Cusum analyses of competency for trainees with unknown and known baseline were comparable (see online supplementary figure S1).
Supplementary figures
Proportion of trainees deemed competent by learning curve cumulative summation (LC-Cusum) method by number of OGDs
The median number of oesophago-gastro-duodenoscopies required to achieve competency for those trainees defined as competent by the learning curve cumulative summation (LC-Cusum) method was calculated. The distribution of the competency run length (crl) for trainees including those with unknown baseline (A) and excluding those with unknown baseline (B).
By moving average analysis, the 95% successful J-manoeuvre rate was achieved by trainees at an earlier stage than the 95% OCR (see online supplementary figure S2).
Trainee extent for incomplete procedures
There were 39 662 incomplete OGDs where the trainee either failed to intubate D2, or managed to intubate D2 but received physical assistance at some point during the procedure. The trainee extent of these incomplete OGDs was: failed intubation 13.1%; oesophagus 23.6%; stomach 44.6%; D1 18.2% and anastomosis 0.4%. Incomplete procedures where the trainee extent was recorded as D2 were excluded as it is impossible to determine at which point during the procedure the trainee required physical assistance. Further analysis of the trainee extent, restricted to the first 50 procedures or to procedures 51–100, showed that with increasing experience, failed intubation and limitation at D1 became less common while limitation in the oesophagus became more common.
Factors associated with completion
The results from the mixed logistic regression model of factors associated with OGD completion are shown in table 2. In terms of trainee and training factors, the total number of procedures performed by a trainee, trainee specialty, trainee age and lower GI endoscopy experience, especially colonoscopy, were independently associated with OGD completion. The relationship between the proportion of OGDs completed and the total number of procedures performed is shown in figure 4A. Younger age, female gender and the absence of sedation were independently associated with completion. The relationship between the proportion of OGDs completed and patient age is shown in figure 4B. Reported diagnoses of achalasia, malignancy, extrinsic compression or varices were associated with incompletion while polyps and peptic ulcer disease were associated with completed procedures.
Results of mixed logistic regression analysis for factors associated with OGD completion
The relationship between proportion of oesophago-gastro-duodenoscopies (OGDs) completed and total number of procedures performed (A). The relationship between proportion of OGDs completed and patient age (B). The number of procedures variable was binned into eight groups and the proportion of OGDs completed was calculated across these groups. Similarly, the patient age variable was binned into six groups and the proportion of OGDs completed was calculated across these groups.
Validation of data submitted to JETS
There were 41 trainees who performed OGDs between September 2009 and March 2014 at the Queen Elizabeth Hospital Birmingham, according to local reporting software. Thirty-two of these 41 trainees are registered on the JETS database and 24 had submitted data to JETS over the same time period as the recorded data on the local reporting software. There was a high correlation between the number of procedures recorded in JETS and the number of procedures recorded in the local reporting software databases for these 24 trainees over this time period (r2=0.99; p<0.01), but a Bland–Altman plot showed that the local number of procedures was on average 8 more than the JETS number (see online supplementary figure S3).
Discussion
We have analysed the learning curve for attaining competency at OGD by two different methods. Results were reported separately for trainees with and without known baseline level of experience. By moving average method, trainees attained the 95% D2 intubation standard at an average of 187 procedures (unknown baseline trainees included) or at 186 procedures (unknown baseline trainees excluded). By LC-Cusum analysis, after 200 procedures, >90% trainees had attained competency, whether or not trainees with unknown baseline were included. The inclusion or exclusion of trainees with unknown baseline experience made little difference to the results of the competency analyses suggesting the majority of trainees with unknown baseline experience were at the beginning of their learning curve. This is the largest study to date analysing the learning curve of competency acquisition for OGDs both in terms of the number of OGDs and number of trainees.
There is a paucity of published data on OGD learning curves. The most relevant study analysed 11 213 OGDs performed by 135 GI fellows and found that 90% completion was achieved by all trainees after 130 procedures.9 Completion was defined as the passage of the scope beyond the pylorus without physical assistance. This is consistent with the findings from our study. Unfortunately, the study did not go beyond 150 procedures to determine when 95% completion was reached. A small prospective study of 12 trainees showed that successful oesophageal (rather than duodenal) intubation without physical assistance reached 90% after 100 procedures.8
The learning curve to achieve competency at OGD is shorter than for colonoscopy.8 ,11 This supports the notion that colonoscopy is a technically more challenging procedure to perform. The most difficult steps in performing OGD are said to be intubation of the oesophagus and passage of the scope through the pylorus.8 ,22 In keeping with this, we found that the most common site of difficulty resulting in incomplete procedures was the stomach. Failed intubation and problems negotiating the duodenum became proportionately less common with experience beyond 50 procedures, perhaps reflecting early improvement of these specific technical skills. We also showed that the learning curve for performing a J-manoeuvre is shorter than for completion to D2. This is perhaps unsurprising as the J-manoeuvre is a single task; completion to D2 requires a series of successful tasks (eg, oesophageal and pyloric intubation). Trainees tend to begin endoscopy training with OGDs, mastering the necessary motor skills to manipulate the endoscope before progressing to colonoscopy. OGD experience is easier to accrue as the procedure time is shorter and more procedures can be performed per list. The greater technical difficulty of performing colonoscopy compared with OGD, combined with a lower completion standard, may explain the focus in the literature on colonoscopy learning curves.6 ,23
We found that, as for learning any motor skill, OGD completion is strongly associated with the total number of procedures performed. However, unlike colonoscopy, OGD completion was not associated with intensity of training.11 This may reflect difficulty for trainees in finding adequate exposure to colonoscopy lists while there is plentiful exposure to OGDs. It has been shown that procedural motor skills may be best acquired by spaced learning rather than by learning in intensive blocks of time.24 Spacing of training sessions are most beneficial for learning simple motor skills; more complex skills (eg, colonoscopy over OGD) may require more intense training.25 In keeping with this spacing theory, we found no association between OGD completion and breaks from OGD training of >6 months. Unsurprisingly, we found that OGD completion is associated with experience in lower GI endoscopy. Being a nurse endoscopist and of a younger age were inversely associated with completion. It is possible that these trainees have limited exposure to patients and procedures that possess transferable skills to the practice of endoscopy and are inherently more cautious. These theories would require further testing. Although group differences in completion rate are interesting, at present, we do not believe that such findings have any implications for training programmes.
OGD completion was significantly positively associated with younger patient age and female gender. Studies have reported variable associations between gender and tolerance to OGD, while generally showing increased tolerance to OGD with advancing age.26–28 Tolerance to the procedure is not the same as completion however. Procedure-limiting diagnoses may be more common in elderly patients and incomplete adjustment for such interactions in our multivariate model may explain the finding of increased incompletion rates with advancing age. The use of sedation was inversely associated with completion. This is surprising as it has been shown that sedation increases the rate of successful OGD compared with controls in a placebo-controlled trial.29 In patients outside of a trial, those opting for sedation may be less likely to tolerate the procedure due, for example, to increased anxiety, leading to an incomplete procedure. Trainees should therefore not favour sedated patients under the misapprehension that OGDs in such patients are easier to complete.
LC-Cusum curves demonstrate an individual's progress and should be used for formative and summative assessments of competency as well as providing useful feedback to the trainee, trainer and training body. We consider that the LC-Cusum method for determining competency has advantages over the moving average method in the demonstration of an individual's progress along the learning curve, while the moving average method allows easier interpretation of the performance of a large group of trainees. The LC-Cusum is capable of distinguishing the different rates trainees acquire during competency and takes account of the variation inherent in small sample sizes. Nevertheless, a requirement for a minimum total number of procedures is valuable to ensure that a certain level of experience and skill have been acquired,30 especially if no other methods are used to determine competency. Given our findings, we suggest that the current minimum JAG requirement and Australian requirements for 200 OGDs are appropriate.1
The focus of this study has been on OGD completion as a marker of competency. Though this is an important marker, it is important to note that to perform OGD competently, it is necessary to comply with a host of other measures.4 We studied performance of J-manoeuvres but thorough mucosal investigation and pathology recognition are difficult to assess. Another limitation of our study is the reliance on self-reported data. Trainees are required to accept an invitation from a hospital to be able to submit procedure data. The hospital endoscopy training lead and JETS administrator then have access to a trainee's data, and educational supervisors will review submitted data as part of regular appraisal. When a trainee submits an e-portfolio for JAG certification, the training lead is required to validate the content. This allows for some validation of the data submitted. Furthermore, we have shown a strong direct correlation between data submitted to JETS (with small under-reporting) and data recorded in local reporting software for a limited number of trainees (n=24). This is consistent with a previously demonstrated direct correlation between numbers of colonoscopy procedures submitted by trainees to JETS and the number of procedures held in local reporting databases.11 A limitation we have already discussed is most trainees did not have a defined baseline level of experience, as this was not a compulsory data field at JETS registration. Finally, the indication for the OGD may not mandate completion to D2. An example of this would be an OGD for monitoring or treatment of oesophageal pathology such as Barrett’s oesophagus. The OCR of an endoscopy unit in the UK is not an audited quality indicator in contrast to a unit's colonoscopy caecal intubation rate,31 and there is therefore no pressure on an endoscopist to complete an OGD to D2 to fulfil audited standards. A trainer may insist that a trainee does not complete an OGD to D2 if it is felt not to be clinically indicated, even if the trainee were physically able to do this independently. This would have a detrimental effect on the trainee's OCR.
In conclusion, we have determined the learning curve for OGD competency (measured by completion to D2), by two statistical methods using a large national endoscopy training database. The 95% completion target is reached after 186 procedures by moving average method, and 90% of trainees are competent by LC-Cusum analysis after 200 procedures. The JAG competency requirement for 200 procedures appears appropriate.
Acknowledgments
The authors acknowledge the help of Richard Chapman and Peter Rogers of Weblogik Ltd who retrieved data from the JETS e-portfolio for use in this study and Kellie Ward of Queen Elizabeth Hospital Birmingham who retrieved data from the local endoscopy database. The authors also acknowledge all of the trainees who responded to the survey.
References
Footnotes
Contributors The study was conceived and designed by STW and PD. Data retrieval and analysis was conducted by STW and MAM. Further advice on analysis was received from AH and RV. Editorial contributions were made by AH, TI, EAG and RV. Final draft was approved by all authors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.