Consider the following setting:
There's a time dependent biomarker X and the outcome of interest is death. Time 0 (t_0) for each patient is their entry into the study. For simplicity, assume I fit a time-varying Cox model and estimate absolute risk using conditional survival probabilities over a time window w.
The model can be evaluated using the c-statistic which tells us if patient who dies at an earlier time would have a higher predicted risk or not. There are also time-varying measures such as the Incident/Dynamic AUC which compare patients at each event time to those still in the risk set. Along similar lines, we also have C/D AUC. But all of these are relative to time since entry into the study.
It seems to me however none of these measures answer the question I want. My problem is this: I would like to rank individuals based on their risk at the prediction time point. The prediction time point t_p is an arbitrary calendar date. The risk set, then, is patients who are at risk at t_p (as opposed to building risk sets based on t_0). I would have the conditional survival probabilities for these patients. How do I evaluate the discrimination capability of the model? Do I just sample random time points and calculate something like a concordance index? Is there a better way to think about this?
PS: In practical terms, we can think of a transplant waiting list. Then we might want to rank patients based on their survival probabilities at any arbitrary time.
Best Answer
If the proportional hazards (PH) assumption holds, the time-varying aspect of the Cox model is time-varying covariate values (e.g., your biomarker), and an individual can have no more than one event, then this is straightforward.
The hazard function is the instantaneous risk of an event given that an individual has been event-free so far. Under the PH and other assumptions just stated, calendar time is irrelevant to the model. If you are comparing individuals who have survived to some particular calendar date, all you need to do is to compare their relative hazards based on the covariate values on that date. The absolute values of the individuals' estimated survival functions at that calendar date don't matter. They have survived that far, so the hazard is the instantaneous risk.
Thus the hazard ratios (or the linear predictors from which they are calculated) as of the calendar date of evaluation provide a rank-ordering. If your regression model is correct and you plug in the current covariate values, the corresponding linear predictor values rank-order the risks as of that date. Provided that your model is correct, the discrimination quality of the model overall should give the best estimate of the discrimination quality at any calendar prediction date.
The problem in practice is that, with a Cox model, it's only the current values of covariates that enter the calculation. You have no information on how the covariate values (and thus the hazards) will change in the future. You might consider joint modeling of covariates and survival. The
JMpackage comes to mind; other joint modeling approaches are described in the R survival task view.Even when the only time-varying aspect of a Cox model is covariate values, some (like the author of the Python
lifelinespackage) argue on epistemological grounds that you shouldn't even be attempting predictions based on time-varying covariate values. If you have a covariate value for an individual at some future time, then the individual has evidently survived to that time already.More complicated scenarios
If the PH assumption doesn't hold or a Cox model includes time-varying coefficient values in addition to time-varying covariates, the best measure of the model's calibration and discrimination would still be its overall performance, however you evaluated the model. As I recall, although the procedure for incident/dynamic AUC estimates can allow for failure of PH or time-varying coefficients with its Schoenfeld estimator, it doesn't handle time-varying covariate values.
Even ranking those at risk at a certain calendar time becomes difficult with these more complicated models. Without a nice summary like a hazard ratio, you have to choose some other estimate of relative survival. Probability of survival overall doesn't help, as it's 0. Choosing estimated median versus estimated mean survival as the criterion could give different rankings. I suppose you could choose to rank based on estimated survival out to some fixed time beyond the calendar date of evaluation, like 2 years or 5 years. But you still have the problem of not knowing the future course of time-varying covariate values, which might be the most important determinant of survival.
If you want to evaluate a model based on performance as of a particular calendar date at some time in the past, you could consider modeling survival with respect to some fixed calendar date as
time = 0rather than with respect to individual entry times. Those who enter after that fixed date would be given left-truncated entry times via the(startTime, stopTime, status)counting process data format that also handles time-varying covariate data. But that wouldn't allow predictions beyond the last calendar date in your data, unless you were willing to fit a parametric survival model and extrapolate. And parametric survival models don't handle time-varying covariates and left truncation well, as such models need to make assumptions about covariate values over the entire span of time.What you seek is thus fraught with practical and theoretical problems. Keep in mind the alleged words of the great US philosopher LB Berra: "It’s Difficult to Make Predictions, Especially About the Future."