Objective To evaluate the sensitivity and specificity of the York Faecal Calprotectin Care Pathway (YFCCP) and undertake a health economics analysis. The YFCCP has been introduced in support of the National Institute for Health and Care Excellence (NICE) guidance DG11. It is designed to improve the sensitivity and specificity of faecal calprotectin (FC) in discriminating the irritable bowel syndrome from inflammatory bowel disease in primary care.
Design To prospectively evaluate the clinical outcomes at 6 months of the first 1005 patients entering the YFCCP. To develop a cost-consequence model using two comparators: one based on clinical assessment and the C reactive protein/erythrocyte sedimentation rate without using FC, and the second using single testing of the standard FC cut-off.
Setting North Yorkshire primary care practices.
Patients Primary care patients fulfilling NICE DG11.
Interventions The YFCCP.
Main outcome measures Clinical outcome measures from secondary care records.
Results The sensitivity and specificity of the YFCCP are 0.94 (0.85 to 0.98) and 0.92 (0.90 to 0.94), giving a negative and positive predictive value of 0.99 (0.98 to 1.0) and 0.51 (0.43 to 0.59), respectively.
Conclusions The YFCCP overcomes the challenges experienced with FC use in primary care, its efficacy matching initial NICE projections. It is readily incorporated into clinical practice. It should represent the framework on which to increase NICE DG11 implementation nationally.
- clinical decision making
- general practice
- irritable bowel syndrome
Statistics from Altmetric.com
Faecal calprotectin (FC) is recommended by the National Institute for Health and Care Excellence (NICE (DG11)) to support in the discrimination of patients with irritable bowel syndrome (IBS) from those with inflammatory bowel disease (IBD) where colorectal cancer is not suspected.1 FC is a sensitive test and has clear potential for patient benefit and healthcare cost savings.2 However, in primary care, it is relatively non-specific and because the prevalence of IBD is low, FC testing runs the risk of creating numerous false-positive results unless an appropriate testing strategy is employed.3–5 Use of a single-point standard cut-off value (taken to be 50 µg/g) on a single FC test may lead to more rather than fewer patients being referred without improving the diagnostic yield of IBD.6–11
In 2014, we conducted a pilot of a care pathway for the use of FC across five primary care practices in the Vale of York Clinical Commissioning Group (CCG).12 From August 2016, this York Faecal Calprotectin Care Pathway (YFCCP) has been introduced across primary care practices in the four CCGs, serving a population of approximately 800 000 and feeding into York Teaching Hospital National Health Service (NHS) Trust (figure 1). In order to achieve effective roll-out, a comprehensive implementation package was developed with the support of the Yorkshire and Humber Academic Health Sciences Network.
Prior to this time, FC had not been available to primary care in the Vale of York CCG while in the Scarborough and Ryedale CCG, the standard cut-off value of FC <50 µg/g had been used.
In brief, the pathway is available to all patients aged 18–60 years presenting to primary care with lower gastrointestinal symptoms, where colorectal cancer is not suspected, but where there is diagnostic uncertainty as to whether the patient has IBS or IBD. It exists in both EMIS and SystmOne patient management system compatible formats. The decision to enter a patient into the pathway is entirely at the discretion of the general practitioner (GP) and no specific symptom criteria are required. Standard investigations are performed as the GP sees fit (full blood count, C reactive protein (CRP), coeliac screen, thyroid function tests and stool cultures) and, if non-diagnostic, a FC is then requested. The result is typically available within a few days and the laboratory report includes management guidance for the GP in line with the pathway. Clinical biochemistry is authorised to reject primary care samples out with the age range 18–60 years and a secondary care lead monitors gastroenterology referrals for compliance. An agreement was obtained from the CCG to reject referrals not compliant with the pathway, these being returned to primary care with an explanatory letter.
If the FC <100 µg/g, the GP is informed that the patient has a 98% probability of IBS and is directed to provide reassurance and treat for IBS in line with NICE and local guidance or to consider an alternative system diagnosis such as gynaecological or urological disease.13 14 The pathway is safety netted by subsequent GP review, and if at this stage the patient is still symptomatic and aged ≥50 years or had a FC ≥50 µg/g, then routine referral to gastroenterology is directed. If the patient is <50 years and the FC <50 µg/g, a further round of reassurance and IBS-directed therapy is recommended before routine referral to gastroenterology if the patient remains symptomatic at that stage.
If the FC ≥100 µg/g, the test is repeated. For pragmatic reasons and in order not to unnecessarily delay patient management, no guidance is given about the withdrawal of non-steroidal anti-inflammatory drugs. If the repeat FC <100 µg/g, the patient is managed as outlined above. If the repeat FC is 100–250 µg/g, then a routine referral to gastroenterology is requested, and if the repeat FC >250 µg/g, then an urgent referral is requested; this normally prompting a ‘straight to test’ colonoscopy at York Hospital.
The aim of this study was to prospectively evaluate the clinical and cost-effectiveness of the implemented YFCCP.
All FC requests from primary care were identified from a search of a single, secure Laboratory Information Management System (Telepath) database. Patients undergoing monitoring of existing IBD with FC were excluded. Using the Trust Patient Administration System (PAS), subsequent clinical outcomes including endoscopic and radiological investigations, referrals to gastroenterology, upper or lower gastrointestinal surgery, dietetics, gynaecology, urology or pain teams and admissions were recorded. Gastrointestinal investigations in the prior 5 years were also recorded. Since the majority of patients were anticipated to have a FC <100 µg/g and the pathway directs that patients continuing to be symptomatic be referred, it was judged that a 6-month period of follow-up would be sufficient to identify a patient with a false-negative result. Aside from some patients living at the CCG borders, private patients and those who might relocate, it was assumed that the majority of patients would be referred to York Teaching Hospital NHS FT.
After storage at 4°C, each stool sample was extracted and analysed using Buhlmann CALEX extraction tubes and the EK-CAL Calprotectin ELISA, respectively (supplied by Alpha Laboratories) to determine the FC level at a cost of £23.00.
Data are presented descriptively as median and IQR. Taking the FC cut-off of 100 µg/g as applied within the pathway, the sensitivity, specificity, and negative and positive predictive values (NPVs and PPVs) were calculated with their corresponding 95% CIs. For the purposes of statistical analysis, a simplifying assumption has been used that all patients either have IBS or IBD. IBS therefore includes other functional intestinal diseases, such as idiopathic slow transit constipation and benign anal canal disease. IBD refers primarily to IBD but also includes other organic enteric diseases (OEDs) diagnosed that would require secondary care intervention. In the evaluation, this included colorectal cancer, high-grade dysplastic polyps, adenomas ≥10 mm or >5 subcentimetre polyps. This assumption did not alter the sensitivity or specificity of the YFCCP compared with IBD alone but changed the prevalence from 5% to 8%. Incidental non-enteric disease is excluded from the evaluation.
York Health Economics Consortium (YHEC) developed a decision tree model to determine the cost-effectiveness of the YFCCP. The model was based on a previously developed model used in 2010 for the Department of Health, NHS Purchasing and Supply Agency, Centre for Evidence-based Purchasing (CEP).2 The model used probabilities based on the prevalence of IBD in the evaluated cohort and the sensitivity and specificity of the evidence pathways. Since robust outcome data from FC testing in primary care are not available, the YFCCP was tested against two comparator pathways and varied for five different evidence sources. The first was a non-FC testing pathway and the second used the standard point cut-off and single test as assumed in current NICE guidance (FC <50 µg/g) (figures 2 and 3). A targeted literature search was used to identify the most appropriate comparators for evaluation. A range of resources was used to inform the costs used in the model (table 1).15–18 The costs for first-line IBS medication were made up of 2 months’ supply of loperamide, mebeverine and ispaghula husk. The costs for second-line IBS medication were made up of 2 months’ supply of amitriptyline hydrochloride and linaclotide.
Comparator 1: non-FC testing pathway
Two sets of published data are used. The first from Tibble et al in 2002 gives the sensitivity and specificity for the erythrocyte sedimentation rate (ESR) and CRP testing to identify IBD in a low-risk patient population.19 This is the reference used for the CEP report.2 The second is from a systematic review by Waugh et al in 2013 that was used in NICE DG11.3 20
Comparator 2: single testing standard cut-off FC testing pathway
This pathway assumes the GP has assessed the patient using a single test and the suggested FC cut-off (<50 µg/g) as in the current NICE guidance DG11.1 Here, three datasets are used. First, the observed patient data from the YFCCP have been analysed and the sensitivity and specificity calculated based on what would have happened had this cohort been referred according to the NICE pathway. Second and third, to ensure consistency, are the published data from Tibble et al 19 and Waugh et al.3 20
As part of this evaluation, we completed an assessment of clinical outcomes in the 280 patients from the Scarborough and Ryedale CCG who had provided a FC in the 6 months prior to the introduction of the YFCCP.
This has been conducted as a service evaluation and so does not require formal ethical approval.
The first 1005 patients entering the YFCCP have been evaluated (from August 2016 to April 2017). Thirty-four patients who did not complete the pathway before secondary care referral and 18 who were not referred despite direction by the pathway were excluded from the evaluation. A further two patients had not completed clinical work-up at the time of the evaluation and were also excluded.
Of the remaining 951 patients, 63% were women and 47% men. The median patient age was 38 years (IQR 27–48). Indications for FC testing are presented in table 2.
Of the 243 patients with an initially raised FC, 53.0% normalised on repeat either becoming <100 µg/g or falling from >250 µg/g to the 100–250 µg/g range. This gives a distribution of FC results as follows:
FC <100 µg/g 808 (85%) patients.
Raised FC repeated 100–250 µg/g 55 (6%) patients.
Raised FC repeated >250 µg/g 88 (9%) patients.
Median time to repeat the FC when initially >100 was 18 days (IQR 14–29).
FC <100 µg/g (808 patients)
Presumably because of a lack of confidence in or understanding of the pathway, 106 patients with a FC <100 µg/g were referred immediately, before the delivery of any IBS-directed therapy. Within this subgroup, 82% went on to have a colonoscopy, flexible sigmoidoscopy or CT colonography. Only a single case of symptomatic diverticular disease and an incidental neuroendocrine tumour were diagnosed. Of the remaining 702, only 8% of patients were subsequently referred routinely to gastroenterology because of persisting symptomatology. A further 53 patients were referred to urology or gynaecology.
FC 100–250 µg/g (55 patients)
These patients were referred to gastroenterology routinely. In line with the care pathway, any investigations were at the discretion of the gastroenterologist responsible. In total, 82% went on to have a colonoscopy, CT colonography or flexible sigmoidoscopy.
FC >250 µg/g (88 patients)
These patients were referred urgently for ‘straight to test’ colonoscopy or an urgent outpatient review. In total, 90% went on to have a colonoscopy or flexible sigmoidoscopy. Median time from second FC result to ‘straight to test’ colonoscopy or outpatient appointment was 25 days (IQR 20–35).21
Diagnoses are presented in table 3. CRP, a systemic marker of inflammation, was normal (<5 mg/L) in 55% of referred patients who subsequently proved to have IBD.
Sensitivity and specificity of the FC cut-off within the YFCCP
The sensitivity and specificity of FC as used within the pathway are 0.94 (0.85 to 0.98) and 0.92 (0.90 to 0.94), respectively. This gives a NPV of 0.99 (0.98 to 1.0) and a PPV of 0.51 (0.43 to 0.59).
In comparison, sensitivity and specificity of 1 (0.72 to 1) and 0.59 (0.52 to 0.65) were obtained on data from 280 patients in Scarborough and Ryedale CCG in the 6 months prior to the introduction of the YFCCP. This gives a NPV of 1 (0.96 to 1) but a PPV of only 0.13 (0.07 to 0.21) when using the standard cut-off value of <50 µg/g on a single stool sample.
YFCCP versus comparator 1: non-FC testing pathway (table 4)
The YFCCP is cost saving and of clinical benefit when compared with both published data for ESR and CRP testing. Here (and in table 5), the data are presented assuming 100% compliance. In the roll-out of the YFCCP, compliance was 85% and so the actual rather than optimal saving amounts to between £60 000 and nearly £100 000 per 1000 patients. By correctly supporting the diagnosis of IBS within primary care, it avoids 100–150 colonoscopies and 140–190 gastroenterology outpatient appointments.
However, there is a trade-off with the Waugh et al pathway, which diagnoses an additional four patients with IBD due to its 100% sensitivity. This is at variance with our observation that the CRP <5 mg/L in over 50% of patients diagnosed with IBD. One explanation for the high sensitivity and specificity data for primary care with no FC testing in the Waugh et al data is that those GPs were better at diagnosing IBS than those in North Yorkshire and that they used ESR and CRP testing as part of the complete diagnostic toolkit. This implies that they were more accurate at referring patients based on symptomatology than if they had relied on ESR and CRP testing alone. This seems unlikely.
YFCCP versus comparator 2: single testing, standard cut-off ≥50 µg/g FC (table 5)
When comparing the YFCCP data based on what would have happened if a single test standard cut-off of ≥50 µg/g cut-off had been used, we see the most dominant result. There is a £150 000 saving with 85% GP compliance and the avoidance of one unnecessary colonoscopy and gastroenterology outpatient referral for every four patients seen in primary care. A substantial saving is demonstrated compared with the published data for ≥50 µg/g cut-off from Tibble et al with both a saving of £50 000 and clinical benefit. The YFCCP diagnosed nearly 100 additional IBS cases in primary care and an additional 4 IBD cases, avoiding >100 gastroenterology outpatient referrals.
When comparing the YFCCP with the NICE published data for the ≥50 µg/g cut-off from Waugh et al, the intervention arm is more costly, incurring an additional £25 000 and is less effective at diagnosing IBS cases but paradoxically diagnoses slightly more IBD cases per 1000 patients at a cost of £20 937 per diagnosed IBD case. This is difficult to reconcile since the cut-off value is doubled in the YFCCP.
As part of the sensitivity analysis undertaken by YHEC, the prevalence of IBD was varied from a range of 0% to 20%. In so doing, we addressed our earlier assumptions around the inclusion of all OEDs and whether we should have looked exclusively at true IBD (which would have given a sensitivity of 0.98 (0.88 to 1) and a specificity of 0.91 (0.89 to 0.93)). The YFCCP was dominant (cost saving and better health benefits) across all outcomes except at a prevalence of 0%. GP compliance with the YFCCP was also modelled between 0% and 100%, and as soon as 1% is reached, the YFCCP arm dominates (cost saving with better health benefits) the comparator, meaning that when one GP starts to use the YFCCP, the model predicts a saving and more accurate diagnosis. Next, the sensitivity and specificity of the YFCCP was varied between 50% and 100% to test a range of scenarios (recognising the respective 95% CI of 85% to 98% and 90% to 94%). The YFCCP was dominant at all levels of specificity above 75%. Below 75%, specificity incurred cost due to the increase in false-positive referrals, though it remained more effective at diagnosing IBD. At a sensitivity and specificity of 70% and 70%, the cost per IBD correctly diagnosed would be £469.
While NICE DG11 promises significant patient benefit and healthcare savings, there has been, since its publication in 2013, no consistent uptake of FC testing in primary care to support the discrimination of IBD from IBS. Largely, this has been because the evidence base to support effective locally agreed care pathways has not existed. Instead, FC testing has tended to be introduced to address demands on resource in either primary or secondary care rather than to optimise patient care. A simplistic one-sample, single cut-off value approach has been used, which tends to optimise the NPV at the expense of the PPV.6–11 Key to the design of the YFCCP is the safety netted ‘traffic light’ system, underpinned by an uplift of the FC positivity threshold and the repeating of a raised test. In optimising the discrimination of IBD from IBS, most patients now benefit from positive expectant local management or early referral with appropriate urgency. The YFCCP achieves this by accommodating for the way FC behaves in primary care and so delivers a sensitivity of 94% and specificity of 92%. It largely corrects for the preanalytical variability seen in the uneven mixing of FC in the stool and because of the low prevalence of IBD improves the PPV without affecting the NPV. In raising the cut-off value from 50 µg/g to 100 µg/g, patients with a FC of 50–99 µg/g are now safely managed as likely having IBS without any loss of sensitivity. Representing 30% of all patients tested, they would otherwise have been referred to secondary care for invasive, expensive and largely unnecessary investigations. Furthermore, a persistently high FC is now sufficiently specific to justify a ‘straight to test’ colonoscopy with a PPV of 0.54. This allows for the accelerated diagnosis of 78% of patients who prove to have IBD. Early diagnosis of IBD supports the NICE IBD quality standard 81 and is a key objective for this pathway.21 Over 50% of patients diagnosed with IBD had a normal CRP demonstrating that this test cannot be relied on in isolation. Those patients with an intermediate FC have a PPV of 0.45 and are better served by routine gastroenterology review in the first place. For those with a FC <100 µg/g, referral to secondary care remains indicated if the patient continues to be symptomatic. Because of this, we judge that the follow-up period of 6 months in this evaluation has been sufficient to identify any patients with false-negative FC results.
When compared with four of the five databases used in the two comparators of the YHEC model, the YFCCP predicts a substantial cost saving. Arguably, the major cost savings are achievable where FC testing, as a single test with the <50 µg/g cut-off, is already in clinical practice within primary care. When the YFCCP data based on what would have happened if a single <50 µg/g cut-off had been applied and is compared with the NICE published data from Waugh et al, there is an estimated additional cost of £180 000 per 1000 patients.3 Yet the FC cut-off is the same in the two groups. We conclude that this additional cost highlights the gap between what NICE had anticipated and what has been borne out in the real world with the roll-out of FC testing into primary care. This goes some way to explain the lack of consistent implementation nationally and means that FC testing used as NICE currently suggests is actually costing the NHS significantly in terms of patient outcomes, waiting times for treatment and longer waiting lists for gastroenterology outpatient appointments. Nearly all the studies used in the Waugh et al systematic review came from secondary care.3 It could be interpreted that the sensitivity and specificity from this pooled analysis reflects FC testing in an optimum environment. It is noted in the systematic review that Jellema et al had reservations about applying results from specialist care to primary care.4 Now, in so much that the sensitivity and specificity are very similar to those of the NICE published data (93% and 94%, respectively), the YFCCP may represent the ‘optimal’ way of applying FC testing to primary care.3
It is important to recognise that the FC cut-off values in the YFCCP are set in the context of the Buhlmann EK-CAL calprotectin ELISA. This is one of a number of calprotectin assays available to the NHS; indeed, a national audit of use has recently been completed confirming more than four assays and the use of a range of cut-off values although 50 µg/g is generally taken to be the standard. Data from the UK National External Quality Assessments Service External Quality Assessmentscheme have shown wide variation in the results produced by different assays.22 23 However, this variation is not as marked in the range 50–250 µg/g that applies here. While we conclude that the design of the YFCCP is robust, we cannot at present assume that any FC assay applied to it will not affect its sensitivity and specificity. It may be that alternative assays will require alternative cut-off values to be applied for optimal efficacy. However, in view of the sensitivity analysis performed by YHEC, we judge that the YFCCP would continue to demonstrate health and cost benefit.
The successful implementation of the YFCCP has been key to ensuring effective embedding into primary care practice, with GP compliance rapidly achieving 85%. This ensures health and cost benefits. Importantly, we had judged that for the pathway to be effective, it had to fit into and support existing primary care practice and to be intuitive. Aside from the age cut-off and the insistence that cancer was not suspected, the only indication for its use is diagnostic uncertainty. The pathway is sufficiently robust to allow GPs with different degrees of diagnostic confidence to apply it effectively. In support of this, while we had projected from the pilot for a 15% onward referral rate of patients remaining symptomatic despite a reassuring FC, it proved to be only 8% of patients. We conclude that for most patients, reassurance and local measures are sufficient for the management of IBS. The care pathway specifically does not direct the nature of treatment to be delivered to those likely to have IBS apart from providing a link to NICE guidance. Instead, it allows space for local treatment protocols to be applied and indeed could support the sort of dietetic initiative developed in Somerset.7 It does, however, direct the GP to consider alternative system diagnoses such as urogynaecological disease.14 Informal feedback from primary care has been positive, not least, we believe, because the pathway provides a risk-based structure on which to base discussions with the patient about non-specific symptoms that can often be difficult to manage. This explains the rapid uptake and high compliance. Informal patient benefit is also gleaned from a patient survey (online supplementary report).
Supplementary file 1
This evaluation is limited by the absence of a primary care database and the inability to perform a gold-standard investigation on those patients with a FC <100 µg/g. It is likely that a small number of patients will have been referred privately or to neighbouring secondary care providers and so will not have appeared on the Trust PAS. We have judged that a 6-month follow-up of patients has been sufficient to pick up most outlying clinical outcomes and to allow any false-negative patients to present. It is also possible that the YFCCP has ‘displaced’ clinical activity by prompting GP to refer patients through different pathways of care rather than requesting a FC. YHEC have used two models on which to compare clinical outcomes. That using data from this evaluation, but based on what would have happened if a single test standard FC cut-off value of <50 µg/g had been used, demonstrated a similar sensitivity and specificity to the cohort of patients in the Scarborough and Ryedale CCG that had been using FC with a <50 µg/g cut-off in the 6 months prior to the implementation of the YFCCP. We infer from this that the biases considered above do not significantly distort the evaluation findings. The other YHEC outcomes come from the extrapolation of previously published data. While open to criticism, in the context of this evaluation, the comparator data represents the best means available to us of assessing the benefit in terms of healthcare resource and goes some way to explaining some of the challenges that have arisen with FC testing until now. Pleasingly, the YFCCP performs compellingly.
Based on FC request activity from primary care since the introduction of the YFCCP, we estimate that there will be approximately 3700 FC requests per million of population per year. This amounts to ongoing savings of up to £590 000 per million per year. We believe this represents the optimal way in which to use FC in primary care for patient benefit and would strongly encourage its wider implementation.
Significance of this study
What is already known on this topic?
Since the publication of the National Institute for Health and Care Excellence (NICE) guidance in 2013, there has been no agreed approach to faecal calprotectin (FC) testing in primary care.
Using the standard 50 µg/g cut-off, many patients with irritable bowel syndrome (IBS) will have a falsely positive FC.
This has resulted in unnecessary referrals into secondary care.
What this study adds?
The York Faecal Calprotectin Care Pathway (YFCCP) overcomes the challenges experienced with FC use in primary care where the prevalence of inflammatory bowel disease is low.
It optimises the sensitivity and specificity of FC for clinical use in primary care to match those in the initial NICE projections.
It is readily incorporated into clinical practice and intuitive to use.
How might it impact on clinical practice in the foreseeable future?
The YFCCP should represent the framework on which to increase NICE DG11 implementation nationally.
The authors are indebted to the Y&H AHSN for their support of this programme and in particular Julie Oldroyd and Stephen Stericker. The authors would like to acknowledge the work of Sarah Whitehead, John Hutton and Matthew Taylor for their work in 2010, which laid the foundations for the current economic evaluation. The final version is approved by all the authors.
Contributors JT is the guarantor of the article. JT directed the study and wrote the first draft. HH and VH contributed to the YHEC analysis. SM contributed to the patient survey. EM performed a comparator evaluation. DT and AJ provided the evaluation dataset. DT, HH, EM, VH and SM assisted in the preparation of the manuscript.The final version is approved by all the authors.
Funding The YHEC analysis was supported by the Yorkshire and Humber AHSN. Alpha Laboratories Limited has provided financial support, not in this, but another diagnostic accuracy study conducted by York Teaching Hospital NHS Foundation Trust.
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement JT holds the original dataset upon which the study was based.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.