HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Naftel, D. C. Search for Related Content PubMed PubMed Citation Articles by Naftel, D. C.

J Thorac Cardiovasc Surg 1994;107:1528-1529
© 1994 Mosby, Inc.

LETTERS TO THE EDITOR

Do different investigators sometimes produce different multivariable equations from the same data?

University of Alabama at Birmingham
Department of Surgery
Division of Cardiothoracic Surgery
736 Zeigler Research Bldg.
Birmingham, AL 35294-0007

To the Editor:

Do different investigators sometimes produce different multivariable equations from the same data? The answer to the question is "yes" This letter attempts to explain why this is the answer, and asks, "Who cares? "

Performing a multivariable analysis is an important part of many investigations. It requires the coalescence of all the other investigative steps. It is an essential step in producing the information for inferences. Quite in contrast to some who claim that multivariable analysis separates the statistician from the data, ¹ I find that proper multivariable analysis keeps the statistician very close to the data. Yet, admittedly, different equations may result from different analyses of the same data. As a step toward understanding why this per se may not be important, I shall enumerate some of the reasons this may occur.

The production of a "final" equation is actually a process consisting of many steps and many decisions. (The "final" equation in this context refers to the final group of selected risk factors and their associated coefficients, standard errors, and p values.) Each of these steps and decisions has an impact on the final equation. A good and responsible statistician makes these decisions in conjunction with his or her colleagues in the study, be it a laboratory or a clinical investigation.

The following is a partial list of the steps and decisions that may influence the final equation.

1. Differing statistical models
The first opportunity for diversity is in the selection of a statistical model for the analysis. The most commonly used model for a discrete point-in-time event, such as 30-day mortality after an operation, is the logistic regression model. If time until the event (the time-relatedness of the event) is considered, then several other possible models can be used, such as the Cox regression ² or hazard function regression models. ³ The selected model can have a great impact on the determination of the final set of risk factors. However, for the remainder of this discussion, assume that a common model has been chosen by all investigators.

2. Differing approaches to missing data
Clinical data sets are seldom complete. Variables such as ejection fraction or preoperative New York Heart Association functional class are often missing for some patients. Also, unfortunately, it is not always obvious what constitutes missing data. For example some data collection forms are constructed so that if a condition is present then it is checked; otherwise it is left blank. The "blank" can be a problem because one is never sure if a blank means that the condition is not present or whether the coder simply neglected to answer the question. Judgment and an intimate knowledge of the data are necessary to resolve such problems. Once the missing data have been identified, then several classic choices are available for handling the missing data.

One is to include in the analysis only those patients who have complete data. This approach can quickly, and often drastically, reduce the number of patients available for analysis. To add to the problem, it is possible that patients with missing data are different from patients with data. For example, ejection fraction may be more likely to be missing in emergency cases than in elective cases. It is possible to test for these complications and make an adjustment.

A similar approach is to change the subset of patients who are being analyzed at each step of the variable selection process by including all patients who have complete data on the group of variables currently being examined in the stepwise selection process. This process has not been automated (so far as I know) in any computer statistical software. Accomplishing this approach can become tedious as the number of variables being examined increases beyond a small number.

Another approach is to exclude from consideration those variables that are not complete on a large proportion of the patients. This can be unsatisfactory because many potentially key variables may be removed from the analysis. Also, no statistical rules exist for deciding how much missing data is too much.

Yet another approach is to replace missing values for a variable with the mean of the variable. For example, if cardiac output is missing for a patient, then assign that patient a cardiac output that is equal to the mean cardiac output for those patients who have cardiac output present.

Good judgment, combined with experience and medical knowledge, is essential in deciding how to handle missing data for each variable. Different statisticians make different decisions.

3. Differing approaches to variables with minimal information
Although a variable may be collected for most patients, it may have such limited information content that little use can be made of the variable for a multivariable analysis. For example, suppose there is a variable: preoperative insertion of an intraaortic balloon pump. The value for this variable may be "no" for all but a very small number of patients. It may be computationally impossible to estimate the effect of balloon pump use. Different statisticians make different decisions when faced with this dilemma, and these decisions may well affect the final form of the multivariable analysis.

4. Differing approaches to correlated variables
If two variables are very highly correlated then both will not be in the final model. A decision must be made as to which one will be allowed to enter into the model. There may also be combinations of variables that are correlated to a single variable, such as height and weight with body surface area. The statistician must be aware of these correlations to avoid strange models in which one variable is counteracting another correlated variable. In many cases, a nonstatistician expert in the area of study can guide the examination or elimination of correlated variables.

5. Differing approaches to the coding of the data
Even though the same data may be given to several statisticians, great room for diversity in their subsequent coding of the data still exists. The statistician must know the data set intimately to properly code the data in a quantitative manner. He or she must make decisions regarding the ordering of levels of a variable, regarding the possible collapsing of low-frequency levels of a variable, and regarding the construction of specific contrasts of interest. The statistician should consider transformations of variables to better understand and estimate the form of the relationship between a variable and the probability of the event.

Some statisticians tend to change all continuous variables into discrete variables. Other statisticians believe that this is a waste of information and should not be done. The final equation can be importantly influenced by the method of handling continuous variables; for example, age can be entered into the equation building process as a continuous variable or it could be coded as either adult or pediatric or coded some way in between.

All of these coding decisions can greatly affect the final group of selected variables and their coefficients. Different statisticians make different decisions.

6. Differing approaches to apparently incorrect data
No matter how thoroughly a large data set is tested and checked, often there will be incorrect values that the statistician should actively try to identify. A sophisticated statistical analysis does not cure poor quality data. A large sample size also does not cure poor quality data. When the statistician uncovers improbable data then he or she must decide on a course of action, such as deleting the entire patient record or setting the value to missing or trying to determine the correct value. Again the nonstatistician expert is necessary to help identify what is and what is not an improbable value. Without this assistance, the statistician must make his or her own decision as to what is valid and what is not. Different statisticians display different levels of aggressiveness for this activity.

7. Differing variable selection methods and criteria for p values
Several statistical methods exist for choosing a subset of significant variables from a collection of potential risk factors. These methods represent an important part of ongoing statistical research. The methods are not algebraically guaranteed to produce the same set of variables. In fact, the methods can produce quite different equations when the variables have many interrelated correlations. See Draper and Smith ⁴ for an entire text on these methods. Draper and Smith are quick to point out that multivariable analysis should not be an automated process, but should involve experts at every step along the way (chapter 8).

Each of these selection methods uses p values to make decisions. The selection of a specific level of p value is arbitrary, although there are certainly traditional levels that are often used (for example 0.05 or 0.01 or 0.1). Most equation-builders advocate a loosening of traditional p values during the selection process to make sure that no variable that is close misses consideration. Also, several different statistical criteria exist for when to stop the equation-building process. There is diversity among statisticians on the proper methods of selecting the final risk factors.

8. Differing computer resources
Even though different statisticians may use the same software, they may still have to make different decisions based on the power of the hardware. A slow personal computer may necessitate the use of shortcut decisions just to get the job done. In some cases this could result in differing equations between analyses performed on a slow personal computer as opposed to a fast mainframe.

9. Differing appreciations of the science
An understanding of the science of a particular investigation is necessary. Most statisticians are not trained in the biology of the heart, the effects of cardiac drugs, and the effect of surgical variables. The statistician may not be aware of what constitutes a nonsensical result. He or she may not know the interactions of diagnosis, clinical variables, and surgical variables. The statistician needs a nonstatistician expert to interpret and identify the scientific principles involved in the analysis.

The analysis process will occasionally reveal subsets of patients that require careful attention to interactions or even require a separate analysis and final equation. Again, the medical expert should be involved in this scientific thinking.

Are these differences important?
In the hands of experts, these differences are usually not critical because the final inferences will likely be similar For example, one analysis may produce increased body surface area as a risk factor while another analysis may produce increased weight as a risk factor. The final inferences would likely be the same in both analyses.

However, many differences in final equations arise as the result of differing levels of experience and expertise of the individuals on the research team and differing levels of collaboration among the team members. These differences can be important, even to the extent that the final equations are in conflict and final inferences may be flawed. Although the process of a multivariable analysis can be described, it is still possible to incorrectly implement the process.

Summary
The creation of a multivariable risk factor equation is not a simple final step of a research investigation. Rather it is the essence of the research. The multivariable analysis of a group of patients is ideally a collaborative, interactive research process. It requires a knowledgeable statistician and a quantitatively oriented physician engaging in effective continual communication. There are many decisions to make along the way, both medical and statistical.

Two different serious teams will likely produce two different equations. However, the models will likely be similar, with differences resulting from correlated variables, different medical weights of importance assigned to the potential risk factors, and differing statistical methodologic decisions. A serious student of nature will use these different equations to generate inferences from different equations, considering them to be supplementary views of each other leading to more reliable inferences than those obtained from one alone.

References

1. Rothman KJ. Modern epidemiology. Boston: Little, Brown, 1986.
   
2. Cox DR. Regression models and life tables. J R Stat Soc B 1972;34:187.
   
3. Blackstone EH, Naftel DC, Turner ME Jr. The decomposition of time-varying hazard into phases, each incorporating a separate stream of concomitant information. J Am Stat Assoc 1986;81:615-24.
   
4. Draper NR, Smith H: Applied regression analysis. New York: John Wiley, 1981.
   

This article has been cited by other articles:

				V. A. Ferraris, F. H. Edwards, D. M. Shahian, and S. P. Ferraris Risk Stratification and Comorbidity Card. Surg. Adult, January 1, 2008; 3(2008): 199 - 246. [Full Text]

				A. Brunelli and G. Rocco Internal validation of risk models in lung resection surgery: Bootstrap versus training-and-test sampling J. Thorac. Cardiovasc. Surg., June 1, 2006; 131(6): 1243 - 1247. [Abstract] [Full Text] [PDF]

				D. M. Shahian, D. F. Torchiana, R. J. Shemin, J. D. Rawn, and S.-L. T. Normand Massachusetts Cardiac Surgery Report Card: Implications of Statistical Methodology Ann. Thorac. Surg., December 1, 2005; 80(6): 2106 - 2113. [Abstract] [Full Text] [PDF]

				G. L. Grunkemeier and A. P. Furnary Mandatory Database Participation: Risky Business? Ann. Thorac. Surg., September 1, 2005; 80(3): 799 - 801. [Full Text] [PDF]

				R. Jin, G. L. Grunkemeier, A. Starr, and Providence Health System Cardiovascular Study Grou Validation and Refinement of Mortality Risk Models for Heart Valve Surgery Ann. Thorac. Surg., August 1, 2005; 80(2): 471 - 479. [Abstract] [Full Text] [PDF]

				D. M. Shahian, E. H. Blackstone, F. H. Edwards, F. L. Grover, G. L. Grunkemeier, D. C. Naftel, S. A.M. Nashef, W. C. Nugent, and E. D. Peterson Cardiac Surgery Risk Models: A Position Article Ann. Thorac. Surg., November 1, 2004; 78(5): 1868 - 1877. [Abstract] [Full Text] [PDF]

				C. G. Koch, F. Khandwala, N. Nussmeier, and E. H. Blackstone Gender and outcomes after coronary artery bypass grafting: a propensity-matched comparison J. Thorac. Cardiovasc. Surg., December 1, 2003; 126(6): 2032 - 2043. [Abstract] [Full Text] [PDF]

				C. G. Koch, C. M. Mangano, N. Schwann, and V. Vaccarino Is it gender, methodology, or something else? J. Thorac. Cardiovasc. Surg., October 1, 2003; 126(4): 932 - 935. [Full Text] [PDF]

				V. A. Ferraris and S. P. Ferraris Risk Stratification and Comorbidity Card. Surg. Adult, January 1, 2003; 2(2003): 187 - 224. [Full Text]

				G. L. Grunkemeier, K. J. Zerr, and R. Jin Cardiac surgery report cards: making the grade Ann. Thorac. Surg., December 1, 2001; 72(6): 1845 - 1848. [Full Text] [PDF]

				E. H. Blackstone and T. W. Rice Clinical-pathologic conference: Use and choice of statistical methods for the clinical study, ""Superficial adenocarcinoma of the esophagus"" J. Thorac. Cardiovasc. Surg., December 1, 2001; 122(6): 1063 - 1076. [Full Text] [PDF]

				T. W. Rice, E. H. Blackstone, J. R. Goldblum, M. M. DeCamp, S. C. Murthy, G. W. Falk, A. H. Ormsby, L. A. Rybicki, J. E. Richter, and D. J. Adelstein Superficial adenocarcinoma of the esophagus J. Thorac. Cardiovasc. Surg., December 1, 2001; 122(6): 1077 - 1090. [Abstract] [Full Text] [PDF]

				E. H. Blackstone Breaking down barriers: Helpful breakthrough statistical methods you need to understand better J. Thorac. Cardiovasc. Surg., September 1, 2001; 122(3): 430 - 439. [Full Text] [PDF]

				V. Rao, J. Ivanov, R. D. Weisel, J. S. Ikonomidis, G. T. Christakis, and T. E. David PREDICTORS OF LOW CARDIAC OUTPUT SYNDROME AFTER CORONARY ARTERY BYPASS J. Thorac. Cardiovasc. Surg., July 1, 1996; 112(1): 38 - 51. [Abstract] [Full Text]

This Article Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Naftel, D. C. Search for Related Content PubMed PubMed Citation Articles by Naftel, D. C.