J Thorac Cardiovasc Surg 2002;124:11-13
© 2002 The American Association for Thoracic Surgery
Heroes and evidence
Victor A. Ferraris, MD, PhD
Tyler Gill Professor and Chief, Division of Cardiothoracic Surgery, University of Kentucky Chandler Medical Center, C208, 800 Rose St, Lexington, KY 40536-0084 (E-mail vferr2@uky.edu).
Received for publication Oct 9, 2001. Accepted for publication Oct 19, 2001.
 |
Introduction
|
|---|
See associated article on page 20.
Every thoracic surgeon should know about Archie Cochrane. He is as close to a true hero as a physician can get, but I am sure there are those who see him as the devil incarnate. As is apparent from some of the highlights of his career (Table 1), he lived during an exciting time. In the 1930s Professor Cochrane was branded as a "Trotskyite" because he advocated a national health system for Great Britain. His advocacy was tempered by 4 years being held in multiple German prisoner of war camps. He saw soldiers die of tuberculosis and he was never sure what the best treatment was. Believe it or not, he did have some alternate therapies to offer the soldiers in the prisoner of war camps. He could choose between collapse therapy, bed rest, supplemental nutrition, or even high-dose vitamin therapy. A quote from his book
1 sums up his frustration: "I had considerable freedom of clinical choice of therapy: my trouble was that I did not know which to use and when. I would gladly have sacrificed my freedom for a little knowledge."
After the war he continued to lobby for a national health service, and the National Health Service was created in the early 1950s. His experience with the uncertainty about the best treatment for tuberculosis and other chest diseases continued after the war, when he became a researcher in pulmonary disease for the Medical Research Council in Great Britain. In the late 1940s and early 1950s, drug therapy for tuberculosis was introduced and there were even some randomized clinical trials that addressed the issue of the best therapy for tuberculosis. With some encouragement from his friends, Archie embarked on a career that would now be called medical epidemiology and health planning. Still with an interest in tuberculosis, now heightened by the fact that he had contracted the disease, Archie wanted to know the best drug therapy for tuberculosis. There were now drugs available that could treat this disease, streptomycin being the first really effective drug against Mycobacterium tuberculosis. 2 He was a patron of randomized controlled trials (or RCTs, as he liked to refer to them) to test important medical hypotheses. He used the evidence gained from these RCTs to make decisions about the best therapy on the basis of available evidence—the beginning of "evidence-based" practice. He believed that RCTs are the best form of evidence to support medical decision making (so-called "class 1" evidence). Initially he was a voice in the wilderness, but this changed little by little. In 1979 he criticized the medical profession for not having a critical summary, organized by specialty and updated periodically, of relevant RCTs. In the 1980s, a perinatal database of important RCTs was developed at Oxford. In 1987, the year before Cochrane died, he referred to a systematic review of RCTs of care during pregnancy and childbirth as "a real milestone in the history of randomized trials and in the evaluation of care," and he suggested that other specialties should copy the methods used. This led to the opening of the first Cochrane center (in Oxford, United Kingdom) in 1992 and the founding of The Cochrane Collaboration in 1993. Now the Cochrane Collaboration is a repository of RCTs, both disease-specific and specialty-specific. Clinicians and lay persons can go to the Cochrane Web site (http://www.cochrane.org/) and find summaries of all available RCTs on a wide range of medical subjects. From the above discussion, I think it is fair to call Archie Cochrane the "father of evidence-based medicine."
Only one of the thousands of summaries of RCTs found on the Cochrane Collaboration Web site has anything to do with any element of cardiopulmonary bypass (CPB). The Cochrane Heart Group reviewed the evidence that hypothermia reduces stroke after coronary artery bypass grafting performed with CPB. In this issue of the Journal, Bartels and coworkers 3 confirm what the Cochrane reviewers have known for a long time—there are few if any RCTs that provide evidence for the way medicine is practiced with regard to CPB.
In summarizing available medical evidence on a given subject, information retrieval is king. The Cochrane reviewers were able to find 17 RCTs of adequate quality that evaluated postoperative stroke in patients having hypothermic CPB compared with normothermic CPB. Bartels and coworkers 3 found 11 or fewer! It is important to understand how the Cochrane reviewers found these 17 RCTs. One has to realize that only about one third of the world's medical literature appears on large computer databases such as MEDLINE, so simply searching MEDLINE for the RCTs of interest is not sufficient. Computerized databases such as MEDLINE are incomplete, especially in areas of subspecialization like cardiothoracic surgery. The Cochrane reviewers perform an exhaustive search of all available literature, not only MEDLINE, but unpublished trials and so-called "fugitive literature" (eg, government reports, proceedings of conferences, published PhD theses). The average thoracic surgeon has not heard of "publication bias," but the Cochrane reviewers have. They realize that RCTs that have a negative result are less likely to pass the peer-review editorial process into publication than RCTs with a significant treatment effect: so-called publication bias in favor of positive clinical trials. For each of the Cochrane reviews, attempts are made to find unpublished, negative trials to add to the body of evidence about a given subject. By these standards, the search for RCTs in the article by Bartels and coworkers 3 is incomplete.
If information retrieval is king in summarizing available medical evidence on a given subject, then statistical analysis of the retrieved studies (including RCTs) must surely share the throne. It must be obvious that all trials or observational studies that address the same outcomes of a given intervention are not the same. There are almost always subtle differences in study design, sample size, analysis of results, and inclusion/exclusion criteria. The object of comparing multiple observational studies and RCTs on the same treatment outcome is to come up with a single summary estimate of the effect of the intervention. Calculating a single estimate in the face of such diversity may give a misleading picture of the truth. There are no statistical tricks that account for bias and confounding in the original studies. Heterogeneity of the various RCTs and observational studies on the same or similar treatment outcomes are the issues. This heterogeneity makes comparison of RCTs a daunting task, about which volumes have been written. 4
There are at least two types of heterogeneity that confound summary estimates of multiple RCTs: clinical heterogeneity and statistical heterogeneity. Statistical heterogeneity is present when the between-study variance is large, that is, similar treatments result in widely varying outcomes in different trials. This form of heterogeneity is easiest to measure. For example, Berlin and coworkers 5 evaluated 22 separate meta-analyses and found that only 14 of 22 had no evidence of statistical heterogeneity. Three of the remaining 8 comparative studies gave different results depending on the type of statistical methods used for the analysis: the more statistical heterogeneity, the less certain the statistical inferences from the analysis.
Clinical heterogeneity of groups of RCTs that assess similar outcomes is much more difficult to assess. Measurement of treatment outcomes has plagued reviewers who try to summarize RCTs. Many RCTs address similar treatment options (eg, hypothermic CPB vs normothermic CPB) but measure slightly different outcomes (eg, stroke or neuropsychologic dysfunction). For example, the Cochrane Heart Group found 17 RCTs that addressed the effect of CPB temperature on stroke. Only 4 of these 17 RCTs measured neuropsychologic function whereas all 17 measured neurologic deficit associated with CPB. In summarizing the results of multiple RCTs comparing a given treatment, it is necessary to match "apples with apples" when looking at outcomes. It is disturbing that Bartels and coworkers list multiple outcome measures for most CPB interventions. As an example, for the optimal core temperature for CPB, they list 8 outcomes covering the entire spectrum from death to myocardial infarction to stroke to postoperative blood transfusion. In a similar analysis by the Cochrane Heart Group, there was a trend toward a reduction in the incidence of nonfatal strokes in the hypothermic group (odds ratio 0.68 [0.43, 1.05]). Conversely, there was a trend for the number of non-stroke-related perioperative deaths to be higher in the hypothermic group (odds ratio 1.46 [0.9, 2.37]). When all "bad" outcomes (stroke, perioperative death, myocardial infarction, low output syndrome, intra-aortic balloon pump use) were pooled, there was no significant advantage of either hypothermia or normothermia (odds ratio 1.07 [0.92, 1.24]). This suggests clinical heterogeneity among the various RCTs evaluated. There are statistical "tricks" that can investigate and explore the differences among studies, such things as stratification or regression, but none of these evaluations of clinical heterogeneity was used in the study by Bartels and coworkers. 3
Of interest, the Cochrane Heart Group concludes that there is no definite advantage of hypothermia over normothermia in the incidence of clinical events after CPB. This constitutes A-level evidence (multiple well-done RCTs) to support the notion that normothermic and hypothermic CPB have equal efficacy for most outcomes. An expert panel reviewing the Cochrane evidence might suggest that there is good evidence (ie, class I evidence according to the American College of Cardiology/American Heart Association [ACC/AHA] guideline nomenclature) that neither normothermic nor hypothermic CPB results in an increased incidence of perioperative complications. This is an entirely different conclusion from the conclusion that Bartels and coauthors reach about the same intervention, because they suggest that there is little evidence to support the usefulness/efficacy of hypothermia in CPB (their so-called level III). For this reason, I am skeptical of the authors' classification of the level of evidence of the various CPB principles. They state that different randomized trials gave conflicting results for a given principle of CPB and, therefore, that principle was automatically given a scientific level II or level III classification. This is neither the intent nor the letter of the various guidelines published for determining evidence-based interventions. 6 For example, for an intervention such as roller pump versus centrifugal pump to be considered class II by the ACC/AHA guidelines, there would have to be conflicting evidence or divergence of opinion about the usefulness of, or the efficacy of, either intervention. There is neither conflicting evidence nor divergence about the usefulness of either intervention. Both work and there is very little difference between the two. A more reasonable interpretation of the available evidence from randomized trials might be that both interventions (roller pump and centrifugal pump) are equally efficacious. There is almost no information in the Bartels manuscript to back up their contention about classification of the various CPB principles. Without better documentation of the classification of each of the levels of evidence, the reader gets the impression that the authors have an ax to grind or have some bias about CPB interventions.
In truth, I think Archie Cochrane would turn over in his grave if he knew how we worship at the alter of RCTs. Archie realized early on that some things cannot and should not be subjected to RCTs, something that Bartels and coworkers have not acknowledged. Archie thought that an RCT that addressed the efficacy of penicillin was unethical, because half of the study group would be denied the standard treatment. Cochrane cited examples of interventions that would never be subjected to RCTs because of ethical concerns. They include cytologic tests for prevention of cervical carcinoma, pulmonary resection for lung cancer, and diet therapy for phenylketonuria. If he could read the articles by Bartels and associates, I suspect he would include many of their principles of CPB in this category. Is it ethical to deny a high-risk patient the benefit of retrograde cardioplegia to prove efficacy of this type of myocardial protection? Most surgeons would say that an RCT is unnecessary and possibly unethical in this situation.
The Editor and the reviewers struggled long and hard with the manuscript by Bartels and colleagues. I suspect that, in the end, the decision to publish the manuscript was based as much on a desire to emphasize the need for a strong evidence-base to support what we do as thoracic surgeons, as on the scientific merit of the publication. The simple fact is that there are gaps in the scientific content of the manuscript by Bartels and associates. However, these gaps are understandable, possibly unimportant, and do not detract from the important principle that evidence-based medicine is here to stay. If he were living, Archie Cochrane would enjoy reading the Bartels manuscript, if for no other reason than the manuscript suggests that we have come full circle. The need for evidence-based medicine had its roots in the chest disease of tuberculosis and now, more than 50 years later, these principles have finally returned to the chest in the form of the heart-lung machine.
|
References
|
|---|
- Cochrane A. Effectiveness and efficiency—random reflections on health services. London: Royal Society of Medicine Press Limited, 1971.
- Daniels M, Hill A. Chemotherapy of pulmonary tuberculosis in young adults: an analysis of the combined results of three Medical Research Council trials. Br Med J. 1952;1:1162.
- Bartels C, Gerdes A, Babin-Ebell J, Beyersdorf F, Boeken U, Doenst T, et al. Cardiopulmonary bypass: Evidence or experience based? J Thorac Cardiovasc Surg. 2002;124:20-7.[Abstract/Free Full Text]
- Petitti DB. Meta-analysis, decision analysis, and cost-effectiveness analysis: methods for quantitative synthesis in medicine. 2nd ed. Oxford: Oxford University Press, 2000.
- Berlin JA, Laird NM, Sacks HS, Chalmers TC. A comparison of statistical methods for combining event rates from clinical trials. Stat Med. 1989;8:141-51.[Medline]
- Braunwald E, Antman EM, Beasley JW, Califf RM, Cheitlin MD, Hochman JS, et al. ACC/AHA guidelines for the management of patients with unstable angina and non-ST-segment elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee on the Management of Patients With Unstable Angina). J Am Coll Cardiol. 2000;36:970-1062.[Free Full Text]
Related Article
-
Cardiopulmonary bypass: Evidence or experience based?
- Claus Bartels, Anja Gerdes, Jörg Babin-Ebell, Friedhelm Beyersdorf, Udo Boeken, Torsten Doenst, Peter Feindt, Michael Heiermann, Christian Schlensak, and Hans-Hinrich Sievers
J. Thorac. Cardiovasc. Surg. 2002 124: 20-27.
[Abstract]
[Full Text]
[PDF]
Top
Introduction
References
