Objective To assess the utility and cost of using routinely collected inpatient data for large-scale audit.
Design Comparison of audit data items collected nationally in a designed audit of inflammatory bowel disease (UK IBD audit) with routinely collected inpatient data; surveys of audit sites to compare costs.
Setting National Health Service hospitals across England, Wales and Northern Ireland that participated in the UK IBD audit.
Patients Patients in the UK IBD audit.
Main outcome measures Percentage agreement between designed audit data items collected for the UK IBD audit and routine inpatient data items; costs of conducting the designed UK IBD audit and the routine data audit.
Results There were very high matching rates between the designed audit data and routine data for a small subset of basic important information collected in the UK IBD audit, including mortality; major surgery; dates of admission, surgery, discharge and death; principal diagnoses; and sociodemographic patient characteristics. There were lower matching rates for other items, including source of admission, primary reason for admission, most comorbidities, colonoscopy and sigmoidoscopy. Routine data did not cover most detailed information collected in the UK IBD audit. Using routine data was much less costly than collecting designed audit data.
Conclusion Although valuable for large population-based studies, and less costly than designed data, routine inpatient data are not suitable for the evaluation of individual patient care within a designed audit.
Statistics from Altmetric.com
The quality of care for people with inflammatory bowel disease (IBD) has been the subject of two recent national audits,1 2 and a third is in progress. Measurement of the quality of care through designed audit requires extensive time and resources for data collection. However, with increasing guidelines for the management of care, in recent years there have been increasing numbers of audits undertaken across the UK. At the same time, substantial amounts of administrative data are collected routinely on hospital admissions (about both inpatients and day cases). It is unclear, though, how much of this routinely collected data might be used for large-scale national audit, and whether it would be an effective method of auditing quality of care, as a pointer for in-depth local audit.
For the first and second national IBD audits,1 2 all hospitals in the UK were invited to submit data extracted from hospital records on 20 consecutive admissions with ulcerative colitis (UC) and 20 with Crohn's disease (CD). Following these audits, a working group of professional organisations has published recommended national standards for IBD care.3 These audits have provided us with an opportunity to assess how far the audit can be replicated using central returns, what the costs of this alternative method would be and whether it might provide an effective way of conducting prospective, sustainable, national audit as a high-level measure of the quality of care.
The aim of this study was to assess the potential for routinely collected data to support a national audit of the management of inpatients with IBD. The main objectives were to compare the data collected in the designed UK IBD audit with routinely collected inpatient data, and to compare the costs of the two methods of audit.
We used the first UK IBD audit as the basis for comparison with routine data. All district general and teaching hospitals in the UK were invited to participate, and it covered retrospective data collection for a median of 19 patients per site who were hospitalised with CD or UC in 2006. We obtained the full dataset for the first UK IBD audit and corresponding administrative inpatient data for Wales (Patient Episode Database for Wales),4 England (Hospital Episode Statistics (HES))5 and Northern Ireland (Hospital Inpatient System).6
Matching patient cases across the UK IBD audit and routine data
We used the following anonymised patient identifiers to match patient records from the UK IBD audit with the corresponding routine data: date of admission, year of birth, hospital, sex and specified principal diagnosis. As both the routine data and the designed audit data sometimes contained errors for individual patient identifiers, we incorporated error margins when matching records in order to increase the likelihood of ‘true-positive’ matches while, at the same time, not inflating the number of ‘false-positive’ matches.
These error margins were as follows. For date of admission, we specified an admission date of plus or minus 3 days, as there were some discrepancies of up to 3 days for admissions that occurred around weekends. For location we matched first using hospital codes, then by trust, as we found that for a substantial minority of hospitals in England and a few in Wales, the hospital trust rather than the admitting hospital was specified in the routine data.
For age, we incorporated an error margin of 1 year either side of the year of birth as the UK IBD audit collected only the year of birth, whereas the routine data provided the age of the patient. For principal diagnosis we found that in some cases a more general diagnosis of IBD rather than CD or UC had been specified or coded in the routine data or CD, UC or IBD had been recorded as a secondary diagnosis. Therefore to minimise the risks of ‘false-negative’ matches, we matched first on diagnoses of CD or UC, and second on IBD in any position on the inpatient record.
When the above criteria were used, matching rates for CD were 89% of cases for Wales and Northern Ireland, and 65% for England. For UC, corresponding matches were 90%, 84% and 64%. As the shortfall in matching in England and Wales was attributable largely to a minority of hospitals, we excluded hospitals with successful matches of ≤40%. This generated matching rates for CD and UC, respectively, of 92% and 96% in Wales, 91% and 91% in England and 89% and 84% in Northern Ireland.
The International Classification of Diseases (ICD-10) codes used for CD, UC and IBD were, respectively, K50, K51 and K50–K52. Episodes of care were record linked to generate inpatient spells and diagnostic codes were based on diagnoses recorded across the inpatient admissions. Details of the data items in HES that were used to compare the UK IBD audit and the administrative inpatient data are provided in the footnotes to table 1.
The second UK IBD audit, conducted during 2008, was used to estimate the cost of providing designed national audit data. This exercise was limited to staff costs, because of expected difficulties encountered by trusts in estimating non-staff resources used in audit activities. Previous studies, however, have shown that non-staff costs represent only a small percentage of total audit costs.7 8
A questionnaire was sent by email to all local audit leads, asking them to identify all staff who contributed to the audit, to provide details of the title and grade of each contributing staff member and to estimate the time spent by each contributor on any aspect of audit activity. Responses were requested at trust level and anonymity was assured.
Where staff grades were provided, hourly rates including on-costs and overheads were taken from Curtis.9 Where job titles were provided without an associated grade, a search was made on the online National Health Service (NHS) job website,10 to identify advertised posts which most closely resembled the stated job title. Hourly rates were assigned to these posts by matching them with the nearest salary band in Curtis.9
Unlike the designed audit, the routine data replication required start-up costs such as capital equipment, preparatory work and staff training. Therefore, two separate analyses of the costs of the routine data replication are provided. The first analysis shows costs as they were incurred in the study. The second analysis reports projected costs of running subsequent routine data audits assuming that no additional staff training is required and that other resources required are already in place. For further details of the methods used for the costing of the routine data audit see the footnotes to table 2.
Using the matching criteria described above, the UK IBD audit data items were compared across designed audit and routine administrative inpatient data (table 1 and figure 1). Although there was variation between countries, there were very high matches (90–100%) for mortality information (both for deaths that occurred in hospital as well as for those that occurred after discharge); surgery; dates of admission, surgery, discharge and death; patient age and sex; principal diagnosis; and hospital/trust.
There were lower matching rates (typically 65–89% but sometimes >90%) for primary reason for admission; source of admission; and recording of colonoscopy or sigmoidoscopy for UC. Matching rates were lower still for specialty of care in the first 24 h and for comorbidities, and were very low or even zero for indications for surgery; specialism of the operating surgeon; and for postsurgical complications. Figure 1 illustrates the consistency across England, Wales and Northern Ireland, both for CD (figure 1A) and for UC (figure 1B).
The overall matching rates for all recorded major comorbidities for CD and UC were, respectively, 47% and 50% in Wales, 28% and 38% in England and 22% and 27% in Northern Ireland (table 1). Of individual comorbidities, matching rates were 50–100% for diabetes, 25–62% for ischaemic heart disease and renal failure and <25% for chronic obstructive pulmonary disease, stroke, liver disease and peripheral vascular disease (table 1).
We were not able to match many of the other detailed data items collected in the UK IBD audit, largely because the data items are not collected by routine data systems or because of the limitations of ICD and Office of Population Censuses and Surveys disease and procedure coding for specialised IBD-specific symptoms, procedures and complications.
Designed audit costs were obtained from 48 returned questionnaires covering 54 of 222 sites which submitted data to the national audit. This represents an overall response rate of 24.3%. A comparison of responding and non-responding sites showed the sample to be over-represented by sites treating a higher number of inpatients with IBD (responding sites mean (SD) total admissions = 204.2 (212.4) vs 143.0 (149.4) in all audit sites reporting total admissions). The sample was also slightly over-represented by sites which submitted data on higher numbers of patients to the national audit (mean (SD) number of patients submitted in responding sites = 34.1 (7.8) vs 30.6 (10.6) in sites which had submitted at least one patient to the audit). Three questionnaires were incomplete and were excluded.
Designed data audit staff costs ranged across trusts from £902 to £10 898 (mean=£2827, SD=£2070). Costs per patient ranged from £24 to £193 (mean=£76, SD=£44). These suggest that the cost for all trusts which participated in the national audit was over £500 000.
For the routine data audit replication, table 2 ‘recorded costs for initial audit’ shows costs as incurred (including preparatory work and training), and table 2 ‘estimated cost of subsequent audits’ estimates costs that would be incurred if the same audit team were to carry out subsequent audits (assuming no preparatory work, reduced capital costs and use of trained staff). Total costs were respectively £22 650 in the initial audit and £15 071 in subsequent audits.
We found high matching rates between the UK IBD audit and routine administrative inpatient data for a small subset of basic important information that is typically collected in designed audits. This includes mortality; major surgery; dates of admission, surgery, discharge and death; principal diagnoses; and sociodemographic patient characteristics including age, sex and social deprivation and hospital/trust.
We found lower matching rates for some further data items, including source of admission, primary reason for admission, some comorbidities and recording of colonoscopy or sigmoidoscopy. Coding of endoscopy may be omitted when the procedure is undertaken in the context of a more complex intervention(s) for severe IBD, which was more typical of the patient case mix in the UK IBD audit.
Routinely collected administrative data were substantially or largely incomplete or did not include most detailed information collected in the UK IBD audit. These include symptoms on admission, detailed disease history, many comorbidities, disease severity, imaging, dietetic details, details of most therapeutic interventions, indications for surgery, details of the surgeon, details of many surgical complications and details of discharge arrangements.
The anonymised patient identifiers made available in the designed UK IBD audit lacked discriminatory power for matching cases, particularly in the event of transcribing errors in either the routine or the designed audit data. For future studies, additional or more discriminatory anonymised patient identifiers would be recommended.
We employed a matching method that incorporated error margins to overcome this lack of discriminatory power. This would have included some ‘false-positive’ matches as a trade off for the improvement in ‘true-positive’ matches. The successful matching rates between the UK IBD audit and the routine data may therefore have been, in reality, slightly lower. This shortfall in matching naturally lowers the observed accuracy of the comparisons of data items between the UK IBD audit and the routine data.
In England, in particular, the trust, rather than the hospital to which the patient was admitted, was recorded on the routine data for some hospital admissions. As trusts can comprise many hospitals, it is important that the recording of hospital codes should be improved.
An important outcome measure that was collected by the designed UK IBD audit, and can be measured accurately through administrative data, is case death. Other outcome measures that are collected in many designed audits and that can potentially be assessed using record linkage of routine data include re-admission rates and infection rates for major infections such as Clostridium difficile and methicillin-resistant Staphylococcus aureus; although these were not assessed in this study.
We found that administrative data are not suitable for detecting more subtle deficiencies in hospital care—based on more specialised outcome measures—such as adherence to disease management guidelines, postoperative and therapeutic complications. However, a potential advantage of administrative inpatient data is that it should cover all admitted patients, whereas some patient cases may be excluded from clinical audit; if, for example, the notes are unavailable. As administrative inpatient data were used to inform the sampling frame of the UK IBD audit, it should not have been subject to biases in patient inclusion that can affect some audits.
The case fatality rate for IBD was 1.2% overall, which is too low for discriminatory comparisons across hospitals. Record linkage of routine inpatient data can be used to identify variation in case death across hospitals for more common, high-risk conditions. The length of time required depends heavily on the hospital admission rate and the case fatality rate for the disease or condition being studied. However, it is important that comparison of mortality across hospitals should be made carefully, because of variations in patient disease severity and case mix that often require adjustment. As we have found in this study, many comorbidities are not recorded comprehensively on inpatient data during single admissions. Record linkage to previous admissions and to alternative data sources—such as primary care data, cancer registry data and operational systems—has the potential to measure comorbidities more comprehensively. Unlike routine administrative inpatient data, however, these more specialised data sources often do not cover national or large regional populations.
The costs to NHS trusts of providing designed data to the UK IBD audit were much greater than the costs of carrying out the audit using routinely collected data alone, but the conclusions that can be drawn will be much more limited.
Many studies have demonstrated the considerable value of routine inpatient data for population studies, including studies of disease incidence, outcomes and risk factors.11,–,18 This study confirms our previous work that identified the limitations of these data when analysed at the level of the individual patient or healthcare professional.19 The weaknesses result from a lack of detailed clinical content, the process of data extraction from poor inpatient records, the use of coding classifications that lack the granularity of a detailed clinical terminology and a lack of the clinical engagement necessary to ensure rigorous data validation.20
This study was an attempt to demonstrate the wider potential of HES data, and to harness national audit as a driver to improve clinical engagement in data collection and validation at the point of care. However, we conclude that if prospective, continuous, sustainable national audits are to be conducted using routinely collected data in place of designed audit data, much change, effort and leadership will be needed to ensure that the data are collected and coded in sufficient detail and with accuracy.
The authors are grateful to the Health Information Research Unit (HIRU), School of Medicine, Swansea University, for preparing and providing access to the project-specific linked datasets from the Secure Anonymised Information Linkage (SAIL) system, which is funded by the Wales Office of Research and Development. The authors are also grateful to the following for various advice and help: Dr Keith Leiper and other members of the UK IBD audit steering committee; Calvin Down, Valerie Porter and Jane Ingham (Royal College of Physicians); Professor Michael Goldacre and Leicester Gill (Unit of Health-Care Epidemiology, University of Oxford); Professor Ronan Lyons, David Ford, Dr Sarah Rodgers, Caroline Brooks, Steven Macey, Tracey Hughes, Rohan D’silva, Jean-Philippe Verplancke (all School of Medicine, Swansea University); Dr Gareth John (National Public Health Service for Wales); Christine Kennedy and Rachel Stewart (Department of Health, Social Services & Public Safety, Belfast); Linda Baumber and Linda Davies (Singleton Hospital, Swansea); and UK IBD site contacts that responded to questionnaires for the costing sections of the project.
Funding This study was funded by The Health Foundation. The views expressed are those of the authors and not necessarily those of the funding body.
Competing interests None.
Provenance and peer review Commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.