Article Text

Download PDFPDF

Development and roll out of the JETS e-portfolio: a web based electronic portfolio for endoscopists
  1. T Mehta1,
  2. K Dowler2,
  3. B C McKaig3,
  4. R M Valori2,
  5. P Dunckley2
  1. 1Bristol Royal Infirmary, Bristol, UK
  2. 2Gloucestershire Hospitals NHS Foundation Trust, Gloucester, UK
  3. 3New Cross Hospital, Wolverhampton, UK
  1. Correspondence to Dr P Dunckley, Gloucestershire Hospitals NHS Foundation Trust, Gloucestershire Royal Hospital, Great Western Road, Gloucester GL1 3NN, UK; paul.dunckley{at}glos.nhs.uk

Abstract

The JAG Endoscopy Training System (JETS) e-portfolio was designed to provide an electronic log of endoscopic experience, improve the effectiveness of training, streamline the JAG certification process and support the quality assurance of trainers, units and regional training programmes. It was piloted in 2008 with an 82.6% uptake in trainees offered the system. The system was released in the UK in September 2009. Steady adoption across the UK demonstrates the service finds it a valuable tool. In time it will be the only vehicle through which a trainee can achieve certification through JAG to practise independently.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

The national audit of colonoscopy in England, undertaken in 1999 and published in 2004,1 identified an unacceptable level of performance of colonoscopy. The audit showed poor supervision of colonoscopy training and low uptake of training courses. In response to this audit, in 2001 and 2003 the Cancer Action Team of the Department of Health commissioned two training programmes in endoscopy. There was a further commission in 2007 aimed at achieving a self-sustaining position in endoscopy training.

These training programmes have had a major impact on training provision in England and similar initiatives have impacted on the other UK nations. There are now a variety of skills based courses in the UK offered to endoscopy trainees and trainers, administered from the JETS (JAG Endoscopy Training System) website (http://www.jets.nhs.uk). These courses have benefited trainees and trainers by improving individuals' technical and training skills with the resultant impact on training in the 250+ other endoscopy training units in the UK. Further initiatives have been implemented to achieve a sustainable and self-improving national endoscopy training programme.

  1. The JAG accreditation of endoscopy units based on the training domain of the endoscopy Global Rating Scale (GRS) (www.grs.nhs.uk).

  2. The requirement for training units to have a critical mass of trainers that have done a ‘training the trainer’ course (www.jets.nhs.uk).

  3. The e-portfolio for trainees and trainers which forms part of the JETS website.

This paper describes the objectives and current position of the JETS e-portfolio. It explains how the e-portfolio was developed and tested, and describes the national roll out. It concludes with a discussion of how the portfolio will benefit the stakeholders of the endoscopy training community.

Objectives of the e-portfolio

  1. To improve the effectiveness and efficiency of endoscopy training.

  2. To provide the information and a process for JAG certification of trainees.

  3. To support quality assurance (QA) of endoscopy trainers.

  4. To support QA of training units.

  5. To support QA of regional training programmes.

Background

Learning endoscopy is based on acquiring a set of competencies in three domains described by Bloom (1956)2: knowledge, skills and attitudes. Within the knowledge, skills and attitudes framework, the major challenge for trainee endoscopists and their trainers is acquiring and teaching technical skills. Endoscopy skills acquisition is accelerated by several factors3,,8:

  • Environment. Learning is best undertaken in a suitable and stress free environment9; Stott10 suggested time spent creating a positive climate encourages serotonin release, which improves cognitive awareness.

  • Case load. Training lists should be adjusted to the current level of the trainee. Billington11 commented that pacing is key; the optimum is to challenge trainees just beyond their present level of ability. If challenged too much, the trainee gives up; if challenged too little, they become bored and learn little.

  • Structured sessions that include assessment and the creation of a personal development plan. Learning is accelerated by a planned approach to training.

    • Opening discussion. Starting each training session with a discussion, which can include review of the personal development plan, is a good opportunity for the trainer to apply the cognitivist theorist Ausubel's ‘advance organiser’,12 namely an opening statement of current knowledge and intended acquisition of new knowledge.

    • Assessment. The session attains maximal learning when a structured formative assessment of competencies is undertaken, which underpins the creation of learning objectives.13

    • Closing discussion. Constructive performance enhancing feedback, discussion of learning objectives and concluding with a review of what has been achieved completes a structured training episode.14,,17

  • Competent trainers. Trainers who are competent endoscopists and competent trainers.3,,5 78

The JETS e-portfolio is more than a logbook of procedures. It encourages the trainer and trainee to enhance the effectiveness and efficiency of training by facilitating, to varying degrees, the above factors. At its best, a training episode is a partnership (a two way conversation with constructive feedback) between trainer and trainee,45 16 and the e-portfolio encourages the two to work together to make the most of the training opportunity.

At the core of the JETS e-portfolio are data acquisition templates. Data from these templates provides performance outputs at individual (trainer and trainee), local, regional and national level. These outputs present an overview of the quantity and quality of training. This enables assessment of the quality of training at the different levels. Finally, these data, and further functionality, provide the information necessary to meet the new requirements for certification of endoscopists for independent practice—so called summative assessments.

Trainees and trainers are likely to have at least one other portfolio to populate. Multiple portfolios are a consequence of the diverse roles characteristic of a modern health professional. Is there a need for another portfolio?

Endoscopy, in contrast with many other procedure oriented specialities, is performed by a variety of health professionals. In the UK, surgeons, physicians, nurses, radiologists, general practitioners and paediatricians perform endoscopy. All of these groups receive different training and are subject to different methods of determining competence, applied with different degrees of vigour and precision. Patients have a right to expect, and will demand, that specialists are trained and certified to unified standards. More importantly perhaps, we have an obligation to our patients to minimise the adverse impact training might have on their care; thus reducing variance in training and certification is more than an academic aspiration. It is expected that the endoscopy e-portfolio will be able to reduce, and will eventually eliminate, the current variance.

Finally, modern IT systems are increasingly able to ‘speak’ to each other; it should be possible to exchange information between portfolios and other databases. Thus a multitude of portfolios becomes a data definition and IT challenge, not a practical one.

The JETS e-portfolio has been integrated into the JETS website. JETS, and the portfolio within it, are sponsored and administered by the JAG, specifically the QA of Training Working Group (www.thejag.org.uk).

The JETS e-portfolio development

Method

The JETS e-portfolio was created in 2007/8 to support the National Nurse Endoscopist Project.18 The purpose of the project was to commission customised training for nurse endoscopists. A secondary goal was to underpin commissioning with a web based e-portfolio designed to facilitate training. In December 2008, the functionality of the e-portfolio was refined to meet the needs of the entire endoscopist workforce. Design and development of the e-portfolio was supported by six trainees. A pilot was launched across three hospitals in two trusts. There was further refinement based on feedback; a staged national roll out began in September 2009.

Data input

The e-portfolio records the following information:

  • Key performance indicators (KPIs). This includes standard performance data such as caecal intubation rate, comfort scores, sedation and polyp detection rates.

  • Direct observation of procedural skills (DOPS). DOPS is a structured method of assessing competencies (www.thejag.org.uk).

  • Trainee learning objectives (TLOs). Trainers are encouraged to create learning objectives in a free text field when completing a DOPS.

  • Trainee assessment of trainer (TAT). The trainee is asked to grade nine attributes of their endoscopy trainer.

To minimise trainer input, the trainee is responsible for logging on, creating the list and steering the trainer to the relevant fields. The e-portfolio guides both through the requirements, as summarised in table 1. Trainer data entry is password protected. There are four endoscopy list ‘scenarios’:

  • Dedicated training list: this is a list specifically designed to meet the needs of the trainee where the case mix and numbers of cases are adjusted to the experience of the trainee.

  • Ad hoc training list: this is a service list which has not been specifically set up or adjusted for the trainee; not all cases will be suitable for training.

  • Service list: this is a list performed by a trainee who has been certified for independent practice.

  • Summative assessment list: this list is set up with cases suitable for a summative assessment.

Table 1

Data capture recommendations for different list scenarios

In addition, there are data inputs from the endoscopy courses registered on the JETS website. This includes courses attended or delivered, and relevant inputs from the courses, such as DOPS and learning objectives for delegates, and feedback for course faculty.

Data outputs

The data outputs consist of charts and tables which summarise and illustrate the quantity and quality of training. All examples of outputs use real data from trainers and trainees.

The trainee summary page demonstrates their endoscopic experience to date and other data relevant to their learning, including courses, DOPS, TLOs, TATs and certification levels. A quarterly summary is illustrated in figure 1.

Figure 1

Example of an output summarising the trainee's training experience. DOPS, direct observation of procedural skills; ERCP, endoscopic retrograde cholangiopancreatography; OGD, oesophagogastroduodenoscopy; PDP, personal development plan.

Summaries of performance can be created for individual procedures. The key KPIs for the procedure such as caecal intubation rates, polyp detection rates and sedation doses for colonoscopy are displayed quarterly (figure 2).

Figure 2

Example of a trainee colonoscopy key performance indicator summary. DOPS, direct observation of procedural skills; OGD, oesophagogastroduodenoscopy; PDP, personal development plan.

The e-portfolio creates illustrations of acquisition of competencies by displaying DOPS outcomes (figure 3). This visual representation allows an instant assessment of whether the trainee is ready for a summative assessment (greater than 90% level 3's and 4's on the last 3 months' DOPS).

Figure 3

Example of progressive direct observation of procedural skills (DOPS) outcomes for a trainee acquiring colonoscopy skills in the ‘skills during insertion and withdrawal domain’. This trainee is ready for his summative assessment in colonoscopy providing he has reached the required level for his key performance indicators.

For trainers the e-portfolio keeps an ongoing record of anonymous feedback from trainees (figure 4).

Figure 4

Example output from the trainee assessment of trainer questionnaire. This summary indicates the trainer is regarded as being very good at encouraging trainees to reflect on their practice and is also a skilled teacher but there is room for improvement, especially with making the trainee feel relaxed and not rushed, and in the evaluation of the trainee's expertise. ERCP, endoscopic retrograde cholangiopancreatography; OGD, oesophagogastroduodenoscopy.

Individuals have access to the data of those trainees and/or trainers for which they are responsible (table 2).

Table 2

Access levels for data outputs according to stakeholder

The training domain of the GRS requires units to display in each endoscopy room a summary of trainees' certification status. This empowers the nursing team to challenge a trainee if they attempt to perform a procedure they are not certified to do. The e-portfolio creates a summary of the certification status for all trainees in that unit (table 3).

Table 3

Example summary of certification levels

The JETS e-portfolio pilot study

Method

The JETS e-portfolio was piloted by 23 trainees and their trainers for 3 months in Gloucestershire Hospitals NHS Foundation Trust and New Cross Hospital NHS Trust, Wolverhampton. All trainees and trainers were asked to provide feedback on the applicability, acceptability and functionality of the e-portfolio in a post-pilot questionnaire.

Results

The feedback from the pilot was very positive (see box 1). The main concerns related to the double entry of KPI data and compatibility with existing portfolios. There is ongoing work to upload data from endoscopy reporting systems to the JETS e-portfolio, and upload from the e-portfolio to the Intercollegiate Surgical Curriculum Programme and the Joint Royal Colleges of Physicians Training Board e-portfolios.

Box 1 Summary of uptake and feedback from the pilot

Time period: 3 months

Total number of procedures added: 1529

Total number of participating trainees: 23

Average number of procedures per trainee: 66.5 (range 0–446)

Number of procedures added to e-portfolio (% of trainees using the e-Portfolio adding the number of procedures listed)

  • >5 procedures 82.6%

  • >10 procedures 73.9%

  • >20 procedures 52.1%

Total number of endoscopy lists: 301

Average number of lists per trainee: 13.1 (range 0–83)

Total number of DOPS submitted: 97

Average number of DOPS per trainee: 4.22

Feedback from trainees

  • Will you continue to use the e-portfolio? 100% Yes

  • Is the e-portfolio an improvement over your current system? 100% Yes

  • Would you recommend the e-portfolio to your colleagues? 100% Yes

National roll out of the JETS e-portfolio

Method

The national roll out of the JETS e-portfolio began in September 2009. The original intention was to stage the roll out but demand for the e-portfolio was so strong it was made accessible to the entire country in April 2010.

The roll out of the e-portfolio is coordinated by the development team and supported by a regional lead in each deanery who acts as a local champion. The regional lead identifies and supports local training leads to implement the e-portfolio. Contact details of the regional leads can be found in the e-portfolio section of the JAG website. Training leads are encouraged to identify an administrator and a medical and surgical trainee lead.

Feedback indicates that trainee enthusiasm for the JETS e-portfolio is a key factor for adoption within a department. As was intended, trainees encourage their trainers to complete their DOPS on the JETS e-portfolio, create learning objectives and review objectives created by others. This trainee driven approach increases trainer exposure to the e-portfolio and encourages a partnership approach to training.45 14

Adoption of the e-portfolio

Figure 5 illustrates the uptake of the e-portfolio by key users (trainees, trainers and training leads) up to 4 October 2010.

Figure 5

Trainee, trainer and training lead e-portfolio uptake since the national roll out.

Figure 6 demonstrates the total e-portfolio activity, including DOPS, lists and procedures.

Figure 6

Total e-portfolio activity since the national roll out and up to September 2010. The right hand y axis refers to the number of procedures, the left hand y axis refers to the number of direct observation of procedural skills (DOPS) and lists.

Future developments

The development team continue to update and improve the e-portfolio. The following updates are due to be completed by December 2010:

  • Summative assessment protocols and electronic JAG certification process.

  • Additional screens for training programme directors to enable them to review trainees' e-portfolios, the quantity of training delivered in a unit and anonymous trainer feedback.

Below are further updates that are due to be completed in 2011:

  • Automatic upload of data from commercial endoscopy reporting systems.

  • Direct links to the surgical and medical online e-logbooks and e-portfolios.

  • Tabs to record endoscopic ultrasonography, enteroscopy and a range of therapeutic training.

The development team aims to continually refine the e-portfolio and welcomes feedback or suggestions on how to improve the system. Suggestions can be made within the e-portfolio, using the feedback button, or by emailing jets.support{at}endoscopy.nhs.uk.

Trainee certification

A key objective for the JETS e-portfolio is to create a web based process that would streamline the JAG certification. JAG has recently revised the certification process with a greater emphasis on the use of formative DOPS defining when a trainee has demonstrated sufficient competence to undergo a summative assessment. These formative DOPS can be recorded on the e-portfolio and reviewed via summary screens (figure 3). The new JAG certification process begins in January 2011. New trainees will need to record their endoscopic experience in the JETS e-portfolio, as application to JAG will be through this system. This will include the recording of summative DOPS, training lead validation, sign off of the e-portfolio and electronic transmission to the JAG for review. Further details on this process can be found on the JAG website (www.thejag.org.uk).

Discussion

The trainee perspective

Historically, the process of collating and analysing endoscopy data by trainees was time consuming and highly variable. Surgical trainees use the pan-surgical electronic e-logbook; medical trainees had the cumulative sum (Cusum) analysis. Neither tool was adequate to submit an application for JAG certification. From the trainee perspective the key attribute of the JETS e-portfolio is a validated, universally recognised record that provides evidence of competence, which can be used for JAG certification, and annual review of competence progression (ARCP) and record of in-training assessment (RITA). The JETS e-portfolio provides all of the information for these requirements using a simple and effective interface.

Advantages for trainees

  • Web based record: can enter data and review the e-portfolio from any internet access. The portfolio transports easily between organisations.

  • Clear, simple and time efficient data entry.

  • Permanent, printable electronic DOPS record.

  • Produces clear, printable, relevant outcomes for ARCP, RITA and penultimate year assessment, and JAG certification.

  • Creates opportunity for anonymised trainer feedback.

  • Automatically populates a personal development plan and enables review of progress for self-directed learning and guidance for trainers.

The trainer perspective

Undertaking a training list with a new trainee can be a challenging experience and planning their long term progression and development even more so.5 The e-portfolio is designed to make these processes easier providing access to a trainee's endoscopic record, relevant competences (DOPS) and to their recent learning objectives created by others.

Trainers on the JETS system have a trainer's e-portfolio: an ongoing quantitative and qualitative record of the training they undertake. This trainer's e-portfolio can be used for self-directed personal development and for annual appraisal. It is anticipated that in time appraisal of a trainer's e-portfolio will be used to assess trainer expertise.

Advantages for trainers

  • Web based access to their own trainer e-portfolio.

  • Overview of the experience of their trainees, enabling ongoing review of their personal development plan.

  • Helps focus training and guide trainee appraisal.

  • Clear, simple and time efficient DOPS entry with password protected sign off.

  • Produces relevant trainer outcomes for appraisal.

  • Creates anonymous, detailed feedback on training.

The training lead perspective

The e-portfolio provides the training lead with an overview of all the training in their department, to aid delivery of a high performing training unit. It enables monitoring of both training inputs (number and type of lists, completion of DOPS and learning objectives) and training outputs (KPIs and trainee competencies). Trainer activity can be reviewed against their job plan, and trainer expertise judged by trainee feedback and skills progression of their trainees. The e-portfolio provides evidence for key measures in the GRS training domain.

Advantages for training leads

  • Automated, printable list of trainee certification levels.

  • Provides data to ensure trainees attend sufficient training list opportunities.

  • Supports trainee and trainer appraisal.

  • Provides a means of monitoring the performance of trainees who are practising endoscopy independently within the unit.

  • Supports many aspects of the GRS training domain.

The training programme director perspective

There are two main advantages for training programme directors: to support the annual review of trainees and to monitor the quantity and quality of training in the training units in their jurisdiction. The annual review of trainees can be arduous and complicated, with each trainee producing data of performance outcomes in different formats. The JETS e-portfolio standardises data collection and creates appropriate outcomes for assessment at annual review.

Training programme directors have an important responsibility to ensure the quality of training at each unit in their locality. Historically, much of their knowledge of training was via word of mouth. The JETS e-portfolio allows direct comparison of units including:

  • Number and competence of trainees.

  • Quality of training.

  • Number and types of lists, and frequency and outcomes of assessments.

Advantages for training programme directors

  • Standardised data outputs of endoscopic experience and competencies to support annual trainee review.

  • Provides a summary of the training a trainee receives within the region.

  • Ability to monitor whether trainees are receiving an adequate amount of training.

  • Provides a means of monitoring the quality of training across the deanery.

The JAG perspective

The JAG is responsible for certificating trainees to practise independently and is currently relying on paper based submissions. This approach may be unreliable and is time consuming (personal observations). The intention is to modernise the process by using the e-portfolio. Further details on the criteria and timescales for introduction of e-portfolio based certification are available on the JAG website (www.thejag.org.uk).

The JAG has responsibility for the QA of training in the UK through the training domain of the GRS and peer review accreditation visits. The e-portfolio will provide more consistent information to support self-reported scores in the training domain, making assessment at peer review easier and possibly more reliable.

The JAG is currently reviewing the process of peer review accreditation of units. One option is to review data from various sources annually and if there are concerns, an early peer review visit would be triggered. The e-portfolio (by providing an overview of training activity) is likely to be one of the key data sources used for this approach to unit accreditation.

Advantages for the JAG

  • Functionality to streamline and improve certification of trainees.

  • Data source for future unit accreditation.

  • Quality assurance of training courses and regional training programmes.

  • Rich source of information for future research on the impact of educational interventions on performance.

Conclusion

The JETS e-portfolio is more than a logbook of performance. It provides functionality to support training and valuable outputs for those involved with the QA of training. In time it will be the only vehicle through which a trainee can achieve certification to practise independently. It provides the trainer with a quantitative and qualitative overview of their training activity, in a portfolio that can be used for assessing their competence as a trainer. A pilot study indicated that the e-portfolio has high user acceptability, and the steady adoption across the UK demonstrates the service finds it a valuable tool. Once roll out is complete, the e-portfolio will provide a rich source of information about training and how it might be delivered more effectively and efficiently.

References

Footnotes

  • Details on how to set up the JETS e-portfolio are available on the JAG and JETS websites. The development team can be contacted by emailing paul.dunckley{at}glos.nhs.uk.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.