| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
J Thorac Cardiovasc Surg 1997;114:903-910
© 1997 Mosby, Inc.
SURGERY FOR ACQUIRED HEART DISEASE |
Funding for this audit was provided jointly by the British Columbia Provincial Panel on Cardiac Care and The University of British Columbia Faculty of Medicine.
Received for publication April 4, 1997 Revisions requested May 21, 1997 Revisions received June 16, 1997 Accepted for publication June 24, 1997 Address for reprints: Theresa Volk, MBA, Division of Cardiovascular Surgery, Faculty of Medicine, 700 West 10th Ave., Floor C, No. 314, Vancouver, B.C., Canada V5Z 4E5.
Abstract
Objective: The British Columbia Provincial Cardiac Registry collects demographic and clinical data on all patients who undergo cardiac surgery procedures in the province. The purpose of this study was to compare the reliability of data contained in the Registry with data contained in hospital charts. Methods: Registry and hospital charts were compared for 480 cases. Thirty cases were randomly selected for the province's 16 cardiac surgeons. For each case, 10 distinct fields were selected for analysis and classified as consistent, inconsistent, or rejected (data unavailable in one or other source). Results: The overall rate of consistency between charts and the Registry was 86.4%, with an inconsistency rate of 9.9% and a rejection rate of 3.7%. Consistency rates varied significantly across the 10 fields and among the 16 surgeons. Pairwise comparisons of rates between fields indicated that specific field types were problematic and should be targeted for improvement. In addition, pairwise comparisons of rates between surgeons indicated that further education on Registry use is required. Conclusions: Recommendations for database design and management include provision of standard definitions for all fields; education of users; extension of the number of mandatory fields; revision of check-off box fields to yes/no/unsure fields; and collection of data close to the time that it is generated.
The British Columbia Provincial Cardiac Registry (Registry) was created in 1990 to meet documentation, research, reporting, and educational needs of the province's cardiac surgeons and the administrative and planning needs of participating hospitals and the Ministry of Health. The provincial initiative to examine cardiac care services began in 1989 when waiting lists for cardiac surgery grew to unprecedented clinically and politically unacceptable levels and patients had to be referred to the United States for care.
1 In late 1989, the Provincial Advisory Panel on Adult Cardiac Care was created to address cardiac care issues in the province. The advisory panel comprises 16 cardiac surgeons and cardiologists from across the province and a representative from the Ministry of Health. In the fall of 1990, the advisory panel decided to adopt and enhance the method of operative data collection and medical report generation in use by the Vancouver General Hospital cardiac surgeons. In January and February of 1991, the four British Columbia hospitals providing all adult open-heart surgery services for 4 million people joined the Registry.
The Registry is based on a relational database system and is composed of three distinct subsystems: cardiac surgery, angioplasty, and pacemaker/defibrillator surgery. The purpose of this audit was to compare data contained in representative fields of the Cardiac Surgery Database to data contained in hospital charts.
Background of the Cardiac Surgery Database
The Cardiac Surgery Database (Database) is composed of four distinct entry modules: cardiac surgery booking; operative report; waiting list reconciliation; and discharge summary. This audit focuses on data collected in the operative report module. At the time of the cardiac operation, the surgeon completes a 16-page form. The identical format has also been evaluated in a preliminary trial using a laptop computer. The collection format includes more than 200 demographic and clinical fields and uses three methods of data collection: check-off box, multiple choice, and fill in the blank.
Once the surgeon has completed and signed the operative report form, a clerk enters the information into the Database and a program simultaneously creates a printed operative report. This replaces traditional dictated reports, is stored in the patient's medical record, and is distributed to the patient's cardiologist, family doctor, and consultants. Selected data fields from each new and modified record are uploaded on a nightly basis via modem to the central provincial database. This amalgamation of records aids in reporting and statistical analysis.
Purpose of the audit
Users of most medical record systems assume that the information therein is reliable and use the systems accordingly. 2 At the same time, the traditional paper-based patient chart is deemed the "gold standard" in terms of patient information. A search of recent literature reveals that various record system holders are questioning the reliability of their databases and are performing audits to test their accuracy. 2-5
This audit was performed primarily to document the comparative reliability of the Database versus hospital charts, examine consistency rates by field and by surgeon, and provide recommendations for improving data collection methods.
Material and methods
Data were drawn from the Database and compared with information from corresponding charts for 480 patients. Consistency rates were analyzed at both the field and surgeon-specific level, but surgeon-field interaction was not investigated. For each of the 16 cardiac surgeons, 30 patients were randomly selected from their 1994 isolated coronary artery bypass graft (CABG) case loads as follows: for each surgeon a sequential number was assigned to each case, and a random number generator was used to select the appropriate number of cases. An additional sample of five cases was included for each surgeon so that if data on any of the first 30 were incomplete they could be replaced. On the basis of a predicted 10% error rate in clinical databases, a sample size of 30 provides an estimate of the consistency rate per question to within ±3% and an estimate of the consistency rate by surgeon to within ±4%. Also, at approximately 30 minutes per chart, the review could be accomplished within a reasonable time frame.
CABG was selected for study because it is done most commonly. Of the 2553 cardiac operations performed in British Columbia in 1994, 1625 (63.7%) were isolated CABGs. The number per surgeon ranged from 49 to 147, with a mean of 102. In addition, CABG surgery is the subject of a large amount of risk factor analysis and outcomes research, which increased interest in determining Database reliability.
Ten fields from the operative report were selected for audit. All of the fields are found in typical hospital and database records of patients who have undergone myocardial revascularization. A variety of sources in the patient chart were referenced for verification of database entry. Before designing the audit, the author/statistician verified that 10 fields, 16 surgeons, and 30 charts per surgeon would provide a sufficient sample size to determine consistency rates. Furthermore, the 10 fields were chosen to compare the reliability of collecting data in different formats (fill in the blank vs check-off box vs multiple choice) and to test the accuracy of clinical information used to measure severity of patient illness for risk stratification versus patient outcomes. Table I describes the 10 fields selected for the audit, and it details where each field is located in the Database and patient chart.
|
Chart information was not always in a compatible form for comparison with the Database. In such cases, relevant information from the charts was recorded and later, with the help of a cardiac surgeon, transfered into an appropriate format. Table II indicates the conversions used for the left main stenosis (LMS) and ejection fraction (EF) fields. The Society of Thoracic Surgeons National Cardiac Surgery Database Manual for Data Managers was used to guide translation of angina symptoms into numeric classification. 6
|
Table III.
|
2 test was used to determine whether an association existed between the specific field or type of field and consistency rate, and between surgeon and consistency rate. If a difference in consistency was observed between fields and surgeons, sources of differences were investigated by means of a pairwise multiple comparison method often referred to as a Bonferroni correction. 7 Because greater than one comparison is being made, a larger difference is required to be denoted as "significant." A difference was considered significant if it exceeded 
Results
Table IV summarizes the overall results aggregated across all fields and all surgeons. The Database was found to be consistent with the chart in 86.44% of the cases.
|
2 test, resulting in p < 0.001. Pairwise comparisons were also used to evaluate consistency rates between fields. A difference of greater than 7.29% in consistency rates was considered to be significant at the 0.05 level, taking the Bonferonni correction into account. Table V summarizes the consistency rates by field (ranked from the least to most consistent). As well, for each field, a listing of the other fields found to have a significantly different consistency rate is given in the final column of Table V.
|
Table VI provides a breakdown of the consistency rates by type of field. The differences were significant (p < 0.001). Pairwise comparisons suggested that fill-in-the-blank fields had the highest consistency level (95.52%), whereas there was no significant difference between check-off box and multiple-choice fields.
|
|
|
Examination of consistency rates
This audit documented an overall inconsistency rate of 9.9% and a difference in consistency between certain fields, field types, and surgeons. Can the factors contributing to these inconsistencies be determined and modified to increase accuracy of cardiovascular surgical clinical reports and databases? The process of data collection needed to be examined to find out whether inconsistencies arose during data recording or when a clerk entered the information into the Database. A subsequent audit of data entry showed high reliability (consistency rate of 99% at each of the four hospitals). Thus surgeon data collection and entry are primarily responsible for the inconsistencies. Factors that may be important in explaining the differences in the overall rates between fields and surgeons include timing of data collection, accessibility of information, data collection method, and surgeon pattern of practice.
Timing
The time fields (crossclamp time and exit operating room time) are the only ones audited that reflect information collected during the actual operation. These fields have two of the highest consistency rates (95.4% and 95.6%, respectively). Their consistency suggests that data should be captured as close to its original generation as possible. Ideally, information obtained during consults and other studies could be entered directly into the Database or automatically downloaded at the time of the operation. To minimize added time demands on health care professionals, competent patients could be asked to pre-complete some sections of the forms regarding diabetes and other present conditions/illnesses during their surgical consult. The reliability of this change would also need to be audited. These steps would considerably reduce the amount of historical data collection required perioperatively and the time required to generate the operative report.
Accessibility of information
Age data are collected very reliably (95.2% consistency). Age is easily accessible, because it appears on many chart pages along with the patient's birth date. Numeric LVEDP data are captured on the Database form in the same way as age. However, LVEDP is the least reliable (68.33%). LVEDP information is usually recorded within the catheterization report, which may take several minutes to find in the chart, or may be missing, especially for urgent and emergency cases. Surgeons may then estimate the LVEDP from ventricular function and clinical factors, but the reliability of this appears poor. The format used should be modified so that perioperative data entries are readily accessible, and options should be provided to indicate that the requested data are not available during the operation and that the value entered is only an estimate.
Data collection method
The method of data capture seems to influence reliability rates. The check-off box method for collection of information is weak inasmuch as it fails to distinguish "no" from "not sure/not available." In 
Table VI, the fields that are captured by the check-off box method are the most inconsistent, with rates of 13.80% versus 9.48% for multiple-choice and 2.81% for fill-in-the-blank methods. Consequently, the Database would be improved if the system was changed to have a "yes" box, a "no" box, and a "not sure/not available" box.
Numeric data are captured for some of the database fields, whereas subjective interpretations are captured into other fields. Data captured in its simplest form was most reliable and consistent (crossclamp time and exit OR time). A request for an actual number encourages the recorder to reference information in the chart. This view is supported by comparing the consistency rates of the exit OR time entry field (96.63%) with elevated serum creatinine (85.21%) or LVEDP check boxes (68.33%).
Accuracy appears to be encouraged by recording specific values rather than broad ranges and by providing definitions. For example, borderline values for LMS and EF were treated differently by different surgeons. Some indicated "LMS greater than 50%" when the catheterization report stated 50%, whereas others did not. Futher, a data entry form idiosyncrasy did not allow for the capture of LMS range when emergency was designated. This resulted in a high rejection rate of 6% for LMS greater than 50% and 5% for LMS greater than 70%. This has been corrected. The low consistency rate for angina class (76.04%) is probably aggravated by the lack of printed definitions on the form and deterioration in the patient's condition while on waiting lists.
Surgeon pattern of practice
Inasmuch as the reports generated by database entry are "published" and are medicolegal documents, it might be expected that all surgeons would strive for a high degree of accuracy. However, some variation is inevitable and differences in consistency rates were striking (Fig. 1). Some surgeons have practice patterns that provide extensive patient contact and consultation before surgery, whereas others operate largely on patients admitted just before the operation through the emergency department, by hospital-to-hospital transfer, or from out of town. In 1994, the rates of emergency and in-hospital transfer of patients undergoing CABG varied between 29.6% and 69.5% (mean 49.6%) for the 16 surgeons. It must be appreciated that level of completion of Database forms is left to each surgeon's discretion, as with the details included in a dictated report. Reliability of Database entry could be aided by providing training in form completion, providing definitions for all fields, making all important fields mandatory, and simplifying the forms by excluding unimportant fields and data from prior examinations and studies.
It is the responsibility of Database managers and contributors to monitor reliability and make improvements. The users must know how closely the stored information reflects the hospital chart, prior clinical records, and actual patient condition at the time of the operation. Registering false information could have important consequences for individual patients, physicians, academics, and administrators. Our Database stores information at the patient-specific level and can be accessed when subsequent care is required. Invalid information could negatively affect future treatment. Furthermore, in many jurisdictions patients have the right to view their own health records. 8 It could be frightening for a patient to find information they knew was incorrect. In addition, incorrect results could cause undue concern or inappropriate complacency on the part of the patient.
The Database is also accessed for administrative, educational, and research purposes. For these uses it is also of utmost importance that the data contained therein be as reliable as possible. Hospital administrators must make decisions to allocate resources and to plan caseloads, staffing, and the procurement of supplies on the basis of mortality and complication rates, device usage, operating room times, length of stay, and patient risk profiles. Reliable data will minimize the risk of budget overruns and assist with fair distribution of resources based on patient need and effectiveness of treatment. The practice of medicine and difficult decisions on resource allocation cannot be allowed to depend on sources that are not sound.
Recommendations
Auditors of the Manchester Orthopaedic Database found that the completeness of data capture can be improved by providing feedback to users on the aims and objectives of the system and its performance. 2 This suggests that the surgeons in our system would benefit from receiving a copy of their audit results.
Additional recommendations include the following:
A number of these recommendations have been implemented or are in the process of being implemented; forinstance, EF, serum creatinine, and LVEDP now ask for a specific value. Standard definitions for Canadian Cardiovascular Society class of angina I to IVc and American Heart Association class of failure of I to IV are alongside their data fields. Data entry clerks are now directly accessing the blood bank information system to document and enter blood product usage. Finally, in the spring of 1997, comparisons of their individual audit results with the group average were distributed to each surgeon. This will be followed by a meeting of all surgeons to allow presentation of the range of surgeon performance and discussion of the results.
Follow-up studies are planned for later in 1997 after the new Database format has been implemented. These studies will determine whether the aforementioned improvements have increased Registry reliability and are more generally applicable to other constituencies and data collection systems.
Footnotes
Drs. Abel, Hayden, and Tyers are all members of the Division of Cardiovascular Surgery, the Department of Surgery, Faculty of Medicine, The University of British Columbia, Vancouver, B.C., Canada. 
*Vancouver Hospital & Health Sciences Center, Vancouver, B.C., Canada.
**Medical Student, The University of British Columbia, Vancouver, B.C., Canada.
***Royal Columbian Hospital, New Westminster, B.C., Canada.
****St. Paul's Hospital, Vancouver, B.C., Canada.
*****The Faculty of Commerce, The University of British Columbia, Vancouver, B.C., Canada.
References
This article has been cited by other articles:
|
|
 |
|
  E. J. Veen, J. Steenbruggen, and J. A. Roukema Classifying Surgical Complications: A Critical Appraisal Arch Surg, November 1, 2005; 140(11): 1078 - 1083. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. A. Yeung-Lai-Wah, A. Qi, E. McNeill, J. G. Abel, S. Tung, K. H. Humphries, and C. R. Kerr New-onset sustained ventricular tachycardia and fibrillation early after cardiac operations Ann. Thorac. Surg., June 1, 2004; 77(6): 2083 - 2088. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 I. D. Nagtegaal, E. K. Kranenbarg, J. Hermans, C. J. H. van de Velde, and J. H. J. M. van Krieken Pathology Data in the Central Databases of Multicenter Randomized Trials Need to Be Based on Pathology Reports and Controlled by Trained Quality Managers J. Clin. Oncol., April 1, 2000; 18(8): 1771 - 1779. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
