Data Quality
The SICSAG data on activity, interventions and outcomes are collected locally, by clinicians, via a bespoke electronic system called WardWatcher (WW). Data is extracted monthly and sent to Public Health & Intelligence (PHI) via a secure web file transfer system (SWIFT).
A high standard of data quality is essential to ensure the SICSAG database is accurate, consistent and comparable across time, and between hospitals. This will ensure decisions to improve quality of care and service provision at hospital, Health Board and national level are based on correct information.
Data is currently stored on a secure server at PHI and will not be stored on any mobile data storage device. Access is restricted and password protected. A person’s access status is validated and authorised every six months. This is an ongoing programme of work and has no specific end date.
The data quality processes undertaken by SICSAG are incorporated into the following:
At point of data entry
WW data collection tool has been programmed to carry some data quality assurance processes. At point of entry, data collection is validated.
The screenshot below is an example of a validation query that WW generates. This example indicates the validation query generated when the date entered for the unit discharge is before the unit admission date.
Case-note validations
Case-note validations will be undertaken periodically throughout the year by Quality Assurance manager, Regional Coordinator or National Clinical Co-ordinator when on site visits. Data from the admission, history, severity, and Augmented Care Period (ACP) pages of WW are prospectively validated by comparing data in WW with data in the case-notes. Severity data cannot be validated in HDU, as it is not collected.
In the past, Local Audit Coordinators also carried out case note validations, results of which showed high quality data had been demonstrated.
The main outcome measure is the level of agreement. When the data taken from the case-notes is the same as the recordings in WW, the two sources of information are in agreement. When the data taken from the case-notes is different from the recordings in WW, the two sources of information are in disagreement. When information is documented in the case-notes but not recorded in WW it is recorded as not yet entered.
The results of these prospective validations are fed back to the lead audit consultants and/or lead nurses. This information is used to identify any areas where further training or support is required.
Central Validation
Database Linkage
All SICSAG data from 1998 to 2013 have been through a linkage process that aims to match SICSAG Critical Care episodes to Public Health and Intelligence (PHI) and SMR01 data scheme which collects data on all general / acute inpatient and day case admissions. All patients recorded in the SICSAG database should have SMR01 records relating to the same hospital stay. 96% of all SICSAG episodes have been matched to an SMR01 stay. This provides an alternative source of information on hospital, ultimate hospital, discharge dates and outcomes. Where the value of these fields is not documented in SICSAG, it has been overwritten with the value derived from linkage to SMR01.
Missing Data/Data Validation
Six monthly validations are collated and sent out to either the lead audit consultant, or to the local audit co-ordinator. Missing data fields will be queried, and validations carried out on extremes of age and length of stay. Any inconsistencies with the CHI number will also be queried. Any changes to the data are made locally and the data re-extracted.
Validating the APACHE II Diagnosis by case-note validation in ICU and Combined Units (2011)
Validating the APACHE diagnosis code and Chronic Health Points recorded in WW is an essential part of this process. As this case mix tool is nearly 30 years old, SICSAG recalibrated the APACHE II diagnosis using an up to date dataset (last 3-5 years).
A subgroup from the SICSAG Steering Group was set up to validate the APACHE II diagnosis and a pilot site was agreed. A random selection of patients admitted to ICU were identified by the SICSAG analyst within ISD, patient case notes requested from medical records and the data recorded were validated against the information recorded in the case notes.
The subgroup agreed that the data validated within the pilot site was of a very high quality with robust data entry and validation process for the diagnosis and chronic health evaluation and this was evident from the validation exercise.
With the success of the pilot, it has been agreed that this exercise will be rolled out to other units in the future as it would be useful to compare results in units where different members of the team collect data.
Chronic Health Points in ICU and Combined Units (2010)
The APACHE II score is influenced by Chronic Health Points (Past Medical History on the History Page). If they have been entered inappropriately this pushes up mortality predictions and underestimates SMR. The definition of each of these is strict and can be seen when you press the Help button while in the History page in WW.
We reviewed Chronic Health Points entered by all ICU and Combined Units in 2010 for each specific category.
Some units were above 3 standard deviations from the mean for specific Chronic Health Conditions. In each instance, the Lead Audit Consultant was contacted and asked to review their own unit’s data and review unit practice ensuring that this information was, and is recorded with reference to the definitions.
Three of the units identified an unusual case mix which explained why their units were different. The fourth unit identified overestimation in their recording and have taken steps to validate their data retrospectively and ensure that, in future, colleagues entering the data adhere to the definitions.
The SICSAG database should be regarded as dynamic and ongoing validation of data may mean that the data is subject to change.