Evaluating and Updating the IMPACT model to predict outcomes in two contemporary North American TBI cohorts

Abstract

The International Mission on Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury (IMPACT) model is a widely recognized prognostic model applied after traumatic brain injury (TBI). However, it was developed with patient cohorts that may not reflect modern practice patterns in North America. We analyzed data from two sources: the placebo arm of the phase II double-blind, multicenter randomized controlled trial Prehospital Tranexamic Acid for TBI (TXA) cohort and data from an observational cohort with similar inclusion/exclusion criteria (Predictors of Low-risk Phenotypes after Traumatic Brain Injury Incorporating Proteomic Biomarker Signatures (PROTIPS) cohort). All three versions of the IMPACT model - core, extended, and laboratory - were evaluated for 6-month mortality (GOSE=1) and unfavorable outcomes (GOSE=1-4). Calibration (intercept and slope) and discrimination (ROC-AUC) were used to assess model performance. We then compared three model updating methods - recalibration in the large, logistic recalibration, and coefficient update - with the best update method determined by likelihood ratio tests. In our calibration analysis, recalibration improved both intercepts and slopes, indicating more accurate predicted probabilities when recalibration was done. Discriminative performance of the IMPACT models, measured by AUC, showed mortality prediction ROCs between 0.61 to 0.82 for the TXA cohort, with the coefficient updated Lab model achieving the highest at 0.84. Unfavorable outcomes had lower AUCs, ranging from 0.60 to 0.79. Similarly, in the PROTIPS cohort, AUCs for mortality ranged from 0.75 to 0.82, with the coefficient updated Lab model also showing superior performance (AUC 0.84). Unfavorable outcomes in this cohort presented AUCs from 0.67 to 0.73, consistently lower than mortality predictions. The closed testing procedure using likelihood ratio tests consistently identified the coefficient update model as superior, outperforming the original and recalibrated models across all cohorts. In our comprehensive evaluation of the IMPACT model, the coefficient updated models were the best-performing across all cohorts through a structured closed testing procedure. Thus, standardization of model updating procedures is needed to reproducibly determine the best performing versions of IMPACT that reflect the specific characteristics of a dataset.