You are here

Article Text

Article menu
Clinical quality
Inter-endoscopist agreement in diagnosis of Barrett's oesophagus
  1. Glen A Doherty1,
  2. Danny G Cheriyan1,
  3. Jan E Leyden1,
  4. John F O'Dowd2,
  5. Frank E Murray1,
  6. Stephen E Patchett1
  1. 1Department of Gastroenterology, Beaumont Hospital/Royal College of Surgeons in Ireland, Dublin, Ireland
  2. 2Department of Pathology, Bon Secours Hospital/Royal College of Surgeons in Ireland, Dublin, Ireland
  1. Correspondence to Dr S Patchett, Department of Gastroenterology, Beaumont Hospital/Royal College of Surgeons in Ireland, Beaumont Road, Dublin 9, Ireland; patchett{at}iol.ie

Abstract

Objective To assess how interpretation of abnormalities at the oesophago-gastric junction (OGJ) when making a diagnosis of Barrett's oesophagus (BO) varies between endoscopists and to examine the impact of the endoscopy experience on these decisions.

Design/setting Members of the Irish Society of Gastroenterology who regularly perform gastroscopy were invited to participate in a web based image assessment study.

Interventions Questions were posed to ascertain level of endoscopy experience, and participants were asked to indicate the presence or absence of BO in 12 endoscopic images of the OGJ.

Outcome measures Primary outcome was overall level of agreement in responses and relationship to endoscopy experience.

Results The responses of 65 clinicians regularly performing gastroscopy were analysed. In 3/12 images, showing typical long segment BO, there was a strong consensus on the endoscopic diagnosis (>95% agreement). However, agreement was fair to poor (κ for multiple raters, 0.31) on the presence or absence of short BO segments at endoscopy. Minimal differences were observed between experienced endoscopists (individuals with >10 years' endoscopy experience) and less experienced counterparts in the threshold for BO diagnosis. Inter-endoscopist agreement overall was not significantly better within the more experienced group.

Conclusion The study demonstrates low interobserver agreement in endoscopic diagnosis of (short segment) BO, even among experienced endoscopists. Given the costs associated with endoscopic surveillance of BO, prompt efforts to promote consensus diagnosis and improve agreement are required as an important quality improvement measure in this area.

https://doi.org/10.1136/fg.2010.001529

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Evidence of a consistent rise in the incidence of oesophageal adenocarcinoma over recent decades1,–,3 has focused interest on Barrett's (columnar lined) oesophagus (BO), the precursor lesion whose incidence also appears to be on the increase.4 5 The association between BO and cancer has led to the development of recommendations for interval endoscopic screening with the goal of early detection of dysplasia.6,–,8 However, screening comes at a cost, with both physical and psychological morbidity and financial implications both for the individual and health system.9

    The diagnosis of BO relies on visualisation at endoscopy of columnar mucosa above the oesophago-gastric junction (OGJ), with confirmation or corroboration on biopsy samples. Making a confident and correct initial endoscopic diagnosis is clearly important in minimising long term costs and morbidity, although relatively little attention has been given to the quality and reproducibility of endoscopic findings of BO. Those studies that have looked at the area have raised concerns about the operating characteristics of endoscopy in diagnosing BO10 and demonstrate limitations in the reproducibility of reported endoscopic findings in a significant minority of patients.11 There have also been suggestions that the level of endoscopy experience has an impact on the quality of endoscopic diagnosis of BO.12

    In cases where the endoscopist is uncertain about the presence or absence of BO, biopsy findings typically do not add clarity as definitive histological evidence of BO is present in only a small percentage of patients.13 14 While the presence of intestinal metaplasia (IM) may corroborate the diagnosis, reliance on the absence of IM on index biopsy neither excludes the diagnosis nor offers much reassurance to the patient in terms of cancer risk.15 The endoscopist can typically expect to receive confirmation of the presence of non-intestinal columnar epithelium (the significance of which depends entirely on the biopsy site). Thus given the importance of the interpretation of initial endoscopic appearances, we set out to examine the levels of agreement between endoscopists about the presence of endoscopic appearances of BO and sought to evaluate the potential importance of endoscopy experience in determining the threshold for making this endoscopic diagnosis.

    Methods

    Internet based image assessment study

    Invitations to participate in an internet based image assessment study were distributed via email to members of the Irish Society of Gastroenterology (ISG). The ISG membership comprises gastroenterologists in practice and in training, surgeons, other clinicians and researchers with an interest in gastroenterology. The ISG does not issue or endorse specific clinical guidelines on the diagnosis of BO but local practice in Ireland is generally in line with the British Society of Gastroenterology recommendations.16 The email contained a link to a web page which featured a range of questions to ascertain the nature of practice of participating clinicians and their level of endoscopy experience and activity. Twelve selected anonymised static endoscopic images of the OGJ were then available to view on the webpage (taken from patients with a history of reflux symptoms or BO who underwent endoscopy at Beaumont Hospital, Dublin, Ireland during a 6 week period in November and December 2007). Participants were asked to select responses on the web page to indicate an endoscopic diagnosis of ‘normal’ or ‘BO’ for each image. When satisfied with their answers, they were invited to hit a ‘submit’ button which returned the results to an electronic database for analysis. No identifiable electronic link to the identity of participants was stored. The study was performed with the approval of Beaumont Hospital Ethics Committee.

    Statistical analysis

    Comparison of differences in proportions was performed using χ2 tests, performed using Instat/GraphPad Prism Software. Kappa statistics for agreement between multiple raters (for nominal data) were performed according to the method described by Siegel and Castellan.17 A p value <0.05 was considered significant.

    Results

    Seventy-three individuals participated in the study, 65 of whom reported regularly performing gastroscopy as part of their clinical practice; only these responses were analysed. Most participants were physicians (n=57), with a small number of responses from surgeons. The combined participation rate for gastroenterology consultant and specialist registrar (SpR) members of the ISG was 51% (49/97). Approximately half of the respondents were practicing as consultants (32/65, 49%), the remainder working as either registrars or SpRs. Almost half of participants (29/65, 45%) had more than 10 years' endoscopy experience and two-thirds (43/65) reported performing over 250 gastroscopies per annum. The participants viewed 12 digital images of the OGJ and were asked to make a judgement on whether the appearances were normal or consistent with BO. The 12 images can be viewed in the supplementary file (figure S1, available online only; images are numbered P1–P12, consistent with the order in which they were presented).

    There was a high level of agreement between participants (>95% responses) on the presence of endoscopic appearances consistent with BO in three of the 12 images presented (P1, P7, P10). These arguably show ‘classic’ appearances of a long Barrett's segment (>3 cm), with either the presence of a wide area of columnar epithelium visible above the confluence of the gastric folds or islands of squamous mucosa within a columnar segment. Levels of agreement were less for the remaining nine images; the responses are summarised in figure 1. Overall, more than 15% of the participants disagreed with the preferred diagnosis of the group in two-thirds (8/12) of the images. In one case (P9), the responses were split almost equally between those who supported a diagnosis of BO and those calling the appearances ‘normal’. In others, a substantial minority disagreed with the preferred diagnosis. The overall level of concordance between participants across the 12 images was fair to poor, with κ statistic for multiple raters of 0.31 (z=13.92, p<0.001).

    Figure 1
    Figure 1

    Responses of participants (when asked to indicate the presence or absence of Barrett's oesophagus (BO)) to all 12 images. Each bar represents the responses to a single image; the numbers of the images are those show in supplementary figure S1 (available online only) and are numbered in the order in which the images were presented. The bars show the proportion of responses which favoured a diagnosis of BO (black) and normal (white). Non-responses for each image are shown in grey.

    Table 1summarises the findings relating to the level of endoscopy experience; according to consultant versus trainee status, level of endoscopy experience and endoscopy volume for the three images with the highest level of disagreement in responses (where differences related to these factors ought to be most visible). No significant differences were observed in the proportion of consultants (vs trainees), of experienced endoscopists (vs those with <10 years of experience) or of higher volume practitioners (vs those performing less than 250/year) favouring a diagnosis of BO in their responses to images P4 or P9. χ2 statistics reflecting significant differences between groups in responses to image P8 were related to the high ‘non-response’ rate in the consultant/experienced endoscopist groups. When non-responders were excluded, these differences were not significant.

    View this table:
    Table 1

    The responses of participants to images P4, P8 and P9

    Finally, with the aim of further exploring this question, all responses were analysed to ascertain whether the level of agreement was different between individuals within the same group (consultant or trainee endoscopist). The κ statistic for the level of agreement in responses (across all 12 images) within the ‘consultants’ group was κ=0.32 (z=9.81, p<0.001) and showed no difference from the level of agreement for responses within ‘trainees’ taken together as a group (κ=0.29; z=8.85, p<0.001). These findings again suggested that level of endoscopy experience was not a significant factor in determining response.

    Discussion

    BO (columnar lined) is found in 5–10% of patients with chronic gastro-oesophageal reflux symptoms attending for endoscopy.18,–,20 The greatest diagnostic accuracy arises from a combination of visualisation of epithelial columnarisation above the OGJ and histological confirmation or corroboration of this finding.16 While demonstration of the presence of IM on biopsy has previously been considered a necessary part of a definitive diagnosis in certain countries, such as the USA,7 opinion is rapidly shifting closer to the framework accepted in the UK and Europe.21 Sampling variations means that failure to demonstrate IM on biopsy does not reliably exclude BO.16 Indeed, it is increasingly recognised that in BO, columnar epithelium without IM is biologically intestinalised22 and shows similar propensity to neoplastic transformation.15 23 Conversely, IM can be seen in biopsies from the cardia and does not appear to confer the risk of dysplasia and cancer associated with even short segment BO.24 25

    All of this shifts the emphasis back to a correct interpretation of the initial endoscopic findings. Biopsy findings generally only serve as adjunctive to a primary endoscopic diagnosis. Difficulties may arise in the correctly endoscopic identification of BO, particularly in the classification of minor abnormalities at the OGJ which may represent a short BO segment. Despite calls to standardise the endoscopic criteria used to describe the OGJ26 and the way we define and diagnose short segment BO,25 there has been little effort made to critically evaluate ‘real life’ differences in how endoscopists report more subtle endoscopic abnormalities. Undercalling these findings may mean ‘at risk’ patients are missed and overcalls lead to unnecessary worry, expense or an erroneous initiation of surveillance (which, once started, can be difficult to stop).

    This study examined this issue by evaluating levels of agreement between participants in making an ‘endoscopic’ diagnosis of BO using an internet based image assessment tool and looking at the impact, if any, of endoscopy experience on the threshold for making a diagnosis. While levels of agreement between participants were excellent when evaluating images of classic (long segment, >3 cm) BO, levels of inter-endoscopist agreement overall were only poor to fair. While use of a validated endoscopic grading system (Prague C&M criteria) for BO can produce impressive levels of agreement regarding the extent of disease,27 even expert endoscopists disagree in their interpretation of minor abnormalities at the OGJ. The κ values for agreement between expert endoscopists on BO segments <1 cm in length in the Prague study was reported as 0.21.27 This is broadly in line with our observations of agreement between ‘non-expert’ endoscopists, where there was poor consensus in interpretation of similar ‘ultra-short’ lesions. It is clear that what one endoscopist calls an ‘irregular Z line’ represents a sufficient abnormality for others to raise the Barrett's flag. Despite some previous studies suggesting that level of endoscopy experience has an impact on diagnostic accuracy for BO,12 our results do not demonstrate any greater agreement among consultant level/experienced endoscopists (>10 years of practice) than their less experienced counterparts. Instead, our results reflect the findings of others, demonstrating low levels of inter-endoscopist agreement in identification of the OGJ, irrespective of experience and specialist training.28

    There are weaknesses in this study, the principal one being that viewing a single still image may not equate to a dynamic ‘real time’ endoscopic evaluation of the OGJ. However, we observed a low rate of non-responses (where participants failed to offer a ‘diagnosis’ for a particular image), suggesting that both the image quality was adequate and there was adequate ‘visual’ information within the static image for a judgement to be made. We have been exploring using short endoscopic video sequences but difficulties with download speeds and image quality led us to proceed with static images in the first instance. We envisage validating our findings in the future using a web based system to allow participants to view dynamic video footage. The participation rate in this study among gastroenterology consultants and SpRs (the single largest group within the clinical membership of the ISG) exceeded 50%. While a higher participation rate would improve the generalisability of the results, we believe the rate achieved is broadly comparable with other similar studies and does not impact on the substantive findings.

    The challenge which emerges from our results is how to improve inter-endoscopist agreement in the interpretation of subtle abnormalities at the OGJ. While endoscopy experience does not appear to be important, there may be a role for education. While complex endoscopic classification systems are unpopular for use in day to day clinical practice (outside of clinical trials), an improved awareness of existing systems to describe the Z line (such as the ZAP classification) may be useful in selecting those patients more likely to have IM at biopsy.28 Education about these criteria does seem to improve the level of agreement in how endoscopists report appearances at the OGJ.28 29 In the near future, so called ‘multi-modal’ endoscopic imaging of the oesophagus offers the promise of better visualisation and better targeting of biopsies,30 but with the risk that endoscopists may also be left with more things to disagree about. Our own approach has been to encourage extensive photography or video imaging by endoscopists of abnormalities at the OGJ for review with colleagues, ideally in the context of a multidisciplinary team meeting with a consensus diagnosis and management plan arrived at following discussion between endoscopists (as a group) and histopathologists. It has been suggested that columnar segments less <1 cm in length should neither be reported nor biopsied,21 given the lack of diagnostic certainty, low risk of dysplasia and doubtful benefit of surveillance. This is a view to which we are sympathetic and a strong endorsement of such an approach by future national and international practice guidelines might improve the confidence of endoscopists to adopt this approach.

    In combination with other measures to improve the quality of diagnostic practice in BO,31 32 harm can be minimised, patient care enhanced and scarce endoscopy resources used to good effect (with avoidance of inappropriate surveillance procedures). However, it is only with the recognition that we may not all ‘see the same things’ that we are likely to improve endoscopy reporting quality in this area.

    Acknowledgments

    The authors acknowledge the assistance of Mr Michael Dineen, CEO of the Irish Society of Gastroenterology, and of Mr Richard O’Connor, of Curratech Internet Consultancy.

    References

    1. ↵
      1. Blot WJ,
      2. Devesa SS,
      3. Kneller RW,
      4. et al
      . Rising incidence of adenocarcinoma of the esophagus and gastric cardia. JAMA 1991;265:1287–9.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Devesa SS,
      2. Blot WJ,
      3. Fraumeni JF Jr.
      . Changing patterns in the incidence of esophageal and gastric carcinoma in the United States. Cancer 1998;83:2049–53.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Powell J,
      2. McConkey CC,
      3. Gillison EW,
      4. et al
      . Continuing rising trend in oesophageal adenocarcinoma. Int J Cancer 2002;102:422–7.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Caygill CP,
      2. Reed PI,
      3. Johnston BJ,
      4. et al
      . A single centre's 20 years' experience of columnar-lined (Barrett's) oesophagus diagnosis. Eur J Gastroenterol Hepatol 1999;11:1355–8.
      OpenUrlPubMedWeb of Science
    5. ↵
      1. Prach AT,
      2. MacDonald TA,
      3. Hopwood DA,
      4. et al
      . Increasing incidence of Barrett's oesophagus: education, enthusiasm, or epidemiology? Lancet 1997;350:933.
      OpenUrlPubMedWeb of Science
    6. ↵
      1. Sampliner RE
      . Practice guidelines on the diagnosis, surveillance, and therapy of Barrett's esophagus. The Practice Parameters Committee of the American College of Gastroenterology. Am J Gastroenterol 1998;93:1028–32.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Wang KK,
      2. Sampliner RE
      . Updated guidelines 2008 for the diagnosis, surveillance and therapy of Barrett's esophagus. Am J Gastroenterol 2008;103:788–97.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Wang KK,
      2. Wongkeesong M,
      3. Buttar NS
      . American Gastroenterological Association medical position statement: role of the gastroenterologist in the management of esophageal carcinoma. Gastroenterology 2005;128:1468–70.
      OpenUrlPubMed
    9. ↵
      1. Wright TA,
      2. Gray MR,
      3. Morris AI,
      4. et al
      . Cost effectiveness of detecting Barrett's cancer. Gut 1996;39:574–9.
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Eloubeidi MA,
      2. Provenzale D
      . Does this patient have Barrett's esophagus? The utility of predicting Barrett's esophagus at the index endoscopy. Am J Gastroenterol 1999;94:937–43.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Kim SL,
      2. Waring JP,
      3. Spechler SJ,
      4. et al
      . Diagnostic inconsistencies in Barrett's esophagus. Department of Veterans Affairs Gastroesophageal Reflux Study Group. Gastroenterology 1994;107:945–9.
      OpenUrlPubMedWeb of Science
    12. ↵
      1. Padda S,
      2. Ramirez FC
      . Accuracy in the diagnosis of short-segment Barrett's esophagus: the role of endoscopic experience. Gastrointest Endosc 2001;54:605–8.
      OpenUrlPubMedWeb of Science
    13. ↵
      1. Takubo K,
      2. Nixon JM,
      3. Jass JR
      . Ducts of esophageal glands proper and paneth cells in Barrett's esophagus: frequency in biopsy specimens. Pathology 1995;27:315–17.
      OpenUrlPubMedWeb of Science
    14. ↵
      1. Srivastava A,
      2. Odze RD,
      3. Lauwers GY,
      4. et al
      . Morphologic features are useful in distinguishing Barrett esophagus from carditis with intestinal metaplasia. Am J Surg Pathol 2007;31:1733–41.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Kelty CJ,
      2. Gough MD,
      3. Van Wyk Q,
      4. et al
      . Barrett's oesophagus: intestinal metaplasia is not essential for cancer risk. Scand J Gastroenterol 2007;42:1271–4.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Playford RJ
      . New British Society of Gastroenterology (BSG) guidelines for the diagnosis and management of Barrett's oesophagus. Gut 2006;55:442.
      OpenUrlFREE Full Text
    17. ↵
      1. Siegel S,
      2. Castellan NJ Jr.
      . Nonparametric statistics for the behavioural sciences. New York: McGraw-Hill Book Company, 1988.
    18. ↵
      1. Winters C Jr.,
      2. Spurling TJ,
      3. Chobanian SJ,
      4. et al
      . Barrett's esophagus. A prevalent, occult complication of gastroesophageal reflux disease. Gastroenterology 1987;92:118–24.
      OpenUrlPubMed
    19. ↵
      1. Mann NS,
      2. Tsai MF,
      3. Nair PK
      . Barrett's esophagus in patients with symptomatic reflux esophagitis. Am J Gastroenterol 1989;84:1494–6.
      OpenUrlPubMedWeb of Science
    20. ↵
      1. Corder AP,
      2. Jones RH,
      3. Sadler GH,
      4. et al
      . Heartburn, oesophagitis and Barrett's oesophagus in self-medicating patients in general practice. Br J Clin Pract 1996;50:245–8.
      OpenUrlPubMedWeb of Science
    21. ↵
      1. Riddell RH,
      2. Odze RD
      . Definition of Barrett's esophagus: time for a rethink—is intestinal metaplasia dead? Am J Gastroenterol 2009;104:2588–94.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Hahn HP,
      2. Blount PL,
      3. Ayub K,
      4. et al
      . Intestinal differentiation in metaplastic, nongoblet columnar epithelium in the esophagus. Am J Surg Pathol 2009;33:1006–15.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Gatenby PA,
      2. Ramus JR,
      3. Caygill CP,
      4. et al
      . Relevance of the detection of intestinal metaplasia in non-dysplastic columnar-lined oesophagus. Scand J Gastroenterol 2008;43:524–30.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Sharma P,
      2. Weston AP,
      3. Morales T,
      4. et al
      . Relative risk of dysplasia for patients with intestinal metaplasia in the distal oesophagus and in the gastric cardia. Gut 2000;46:9–13.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Sharma P,
      2. Morales TG,
      3. Bhattacharyya A,
      4. et al
      . Dysplasia in short-segment Barrett's esophagus: a prospective 3-year follow-up. Am J Gastroenterol 1997;92:2012–16.
      OpenUrlPubMedWeb of Science
    26. ↵
      1. Armstrong D
      . Review article: towards consistency in the endoscopic diagnosis of Barrett's oesophagus and columnar metaplasia. Aliment Pharmacol Ther 2004;20(Suppl 5):40–7.
      OpenUrl
    27. ↵
      1. Sharma P,
      2. Dent J,
      3. Armstrong D,
      4. et al
      . The development and validation of an endoscopic grading system for Barrett's esophagus: the Prague C & M criteria. Gastroenterology 2006;131:1392–9.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Amano Y,
      2. Ishimura N,
      3. Furuta K,
      4. et al
      . Which landmark results in a more consistent diagnosis of Barrett's esophagus, the gastric folds or the palisade vessels? Gastrointest Endosc 2006;64:206–11.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Wallner B,
      2. Sylvan A,
      3. Janunger KG
      . Endoscopic assessment of the ‘Z-line’ (squamocolumnar junction) appearance: reproducibility of the ZAP classification among endoscopists. Gastrointest Endosc 2002;55:65–9.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Sharma P,
      2. Bansal A
      . Toward better imaging of Barrett's esophagus—see more, biopsy less! Gastrointest Endosc 2006;64:188–92.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Ofman JJ,
      2. Shaheen NJ,
      3. Desai AA,
      4. et al
      . The quality of care in Barrett's esophagus: endoscopist and pathologist practices. Am J Gastroenterol 2001;96:876–81.
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Curvers WL,
      2. Peters FP,
      3. Elzer B,
      4. et al
      . Quality of Barrett's surveillance in The Netherlands: a standardized review of endoscopy and pathology reports. Eur J Gastroenterol Hepatol 2008;20:601–7.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    Footnotes

    • Competing interests None.

    • Ethics approval The study was conducted with the approval of the Beaumont Hospital Ethics (Medical Research) Committee, Dublin, Ireland.

    • Provenance and peer review Not commissioned; externally peer reviewed.