Discussion

Article Outline

• Copyright

Dr Marc R. de Leval (London, UK). I would like to congratulate Dr Al-radi and his colleagues for an important contribution to outcome analysis. The work is a validation study of 2 procedure-adjusted risk stratification methods based both on subjective opinions of a panel of experts. The hospital mortality predicted by the 2 scoring systems is compared with the observed hospital mortality following 13,675 operations performed in a single institution over a 22-year period. Two main findings can be extracted from their analysis.

First, the RACHS-1 categories more consistently represented the probability of hospital deaths compared with the ABC scoring system. We made similar observations in our institution. We assigned the ABC score and the RACHS-1 risk categories to 1085 open cardiac operations performed in the current era. Multiple logistic regression identified RACHS-1 category to be a powerful predictor of mortality, with a P value of <.0001, whereas the ABC score was only weakly associated with mortality, with a P value of .03.

The second finding is that both methods are weak discrimination tools in predicting hospital mortality. The authors claim that it is difficult to expect that knowing little else than the procedure, one can accurately predict the outcome. They imply that much more data, both patient- and anomaly specific, would be required. It will be interesting, of course, to see whether the comprehensive ABC score will be a more effective predictor of outcome. We must accept, however, that it will always be impossible to completely predict outcome, and the question is, how complicated should a risk adjustment be?

If the purpose is to be able to compare institutions or individual surgeons, it is important that patient- and procedure-specific factors do not overwhelm potential institution- or surgeon-specific factors. It would be better to try to understand the reasons for variability between institutions that are not going to be explained by minutely detailed case mix adjustment.

I have 2 questions. The first is why do you think that RACHS-1 is superior to the ABC score system in predicting hospital mortality? Do you think that the concept of complexity, which includes technical difficulty, weakens the power of predicting hospital mortality? Today, many technically challenging procedures, such as an arterial switch operation, carry a very small risk of mortality indeed.

And my second question is have you considered putting the 2 scoring systems together in the same equation to find out whether the combination could increase the power of prediction?

Again, I would like to congratulate you for this study and I thank the Association for inviting me to discuss this work.

Dr Osmon O. Al-Radi (Toronto, Canada). Dr de Leval, thank you very much for your remarks. Regarding the first question, why RACHS-1 is superior, I think the main advantage of RACHS-1 is that the difference between the highest- and the lowest-risk categories is larger than what it is in ABC. A difference in ABC is about 15% between the lowest- and the highest-risk categories, and the spread between the extreme categories is wider in RACHS-1. The other potential cause is that RACHS-1 in some cases incorporates additional factors other than the operation itself. For example, age in coarctation of the aorta is assigned to a higher-risk category if the patient is older. That is not the case of ABC.

Obviously a more comprehensive score such as the Aristotle comprehensive score will add to the discrimination ability of any tool; however, there is a trade-off between simplicity of use and how much data you need to use the score and whether it would be applicable to data that you have already collected and between how powerful the tool is going to be. You have to establish a balance between how complex you want the score and how powerful do you want it to be. So you have to choose a point that satisfies both the discriminating power and simplicity of use.

In regards to your second question, if you put RACHS-1 and ABC in the same model, RACHS-1 comes out as more predictive. It accounts for all what ABC is telling you. So basically ABC would not be significant if you put them in the same model.

Dr Francois Lacour-Gayet (Denver, Colo.). Dr Al-Radi, I have listened with great interest to your presentation. The basic score is the first level of the complexity. It is only a procedure-adjusted complexity, as is RACHS-1. We all know that there are simple Norwood and complex Norwood, simple switch and complex switch. A comprehensive and exhaustive analysis is needed to study individual outcomes.

I will not discuss from a statistical perspective, but intuitively it seems problematic that you ignore in your calculation that there are 4 times the number of patients that could not be analyzed with RACHS-1 compared with ABC.

Finally, constructing a case mix in congenital heart surgery is very challenging. It needs time and attention to detail. We understand that a performance evaluation based on subjective probability and surgical-based knowledge requires a cautious validation. It is in progress. However, today, in absence of validated data in our specialty, if we wait for the data to speak by themselves, there will be only a galactic silence.

Dr Al-Radi. In regards to your first question, there is, again, a balance between how much coverage you want from the scores, whether it covers your entire patient population, and predictive power. You have to establish a balance, again, because if you include patients that have secondary operations, resternotomies, VAD support, that will reduce the predictive power of your score. So, again, it is a balance between how powerful you want the tool to be and the extent of coverage in terms of the procedures that the risk score covers. In regards to your second remark, I have no comment.

Dr Jeffrey H. Silber (Philadelphia, Pa.). I am not a cardiac surgeon but I direct the Center for Outcomes Research at The Children’s Hospital of Philadelphia and teach severity adjustment at The Wharton School of The University of Pennsylvania, and I really see 2 major problems with this study.

The first is that you used fewer variables to describe the ABC score than you did to describe the RACHS-1 score, and it is very elementary to realize that if you have more variables in a model, you will do a better job fitting the data. Why didn’t you fit the ABC score with the same number of variables that you used for the RACHS-1 score? By using fewer variables, you have handicapped the ABC system in your comparisons. The second fundamental problem I see is that you have used different patients to make your comparisons of c-statistics. One of the absolutely essential requirements for comparing severity scores is to use the same patients. By not using the same patients, we really gain very little information as to the comparison between the 2 methods, especially as a larger group of patients were used in the ABC score than the RACHS-1 score. So, not only did you handicap the comparison through your choice of variables, but you also made the comparison meaningless by reporting c-statistics on different populations. I would like to hear your comments on that.

Dr Al-Radi. We did compare RACHS-1 and ABC both as a continuous score and as levels, and we chose the levels for the presentation for the simplicity of the graphs. If you used the continuous score, you would have to use 3-dimensional plots, which I have an example of. The predictive power of ABC did not change whether you used the whole score as a continuous variable or whether you used the ABC as a categorical 4-level variable.

As to your second comment, we also did a sensitivity analysis, including only patients who matched for both scores, and if you do that, the discrimination of the ABC score is somewhat higher but it is still inferior to the RACHS-1.

Dr Silber. Was there a statistical difference between the 2?

Dr Al-Radi. Yes, there was still a statistical significance. But the major point of this presentation is not the comparison between ABC and RACHS-1. I wanted to portray that both scores are short of what would be acceptable as a good method of risk adjustment, and in isolation neither would be adequate for comparing surgeons and institutions. Whether you use RACHS-1 or ABC, you still have to understand that neither is a method that is adequate for complete risk adjustment.

Dr Christo I. Tchervenkov (Montreal, Canada). I would just like to raise the issue of the meaning of validation. Simply, the ABC score was based on the opinion of 50 surgeons from across the world, and because the basic premise of the ABC score is that each patient has a constant complexity no matter where in the world this patient is operated, to what extent do you think that the study using data from a single institution has any meaningful significance as to the question of validation?

If you apply the data from another institution that might have a different performance level, then the conclusions may be completely different. What are your comments or thoughts about that and what is it going to take to validate these scores? It perhaps is going to take the data from multiple institutions across different performance levels, different parts of the world.

Thank you very much.

Dr Al-Radi. Our study only addresses 1 aspect of score validity, which is termed criterion validity or comparing a score to actual data, and obviously because our data were from a single institution, I do not have the ability to generalize it to a multi-institutional database. If a multi-institutional database was available with the outcomes of interest, then it would be very reasonable to reproduce this work with multi-institutional database. So that would be a very good project.