Calculating R Squared on Subsets – Reference Guide for Regression Analysis

r-squaredreferencesregression

I've been looking for a method to calculate $R^2$ on a subset of the samples (a subset of the instances, not a subset of features), and found this answer from Dave. It suggests using the mean of the original samples ($\bar{y}$), rather than the mean of the subset, when calculating the TSS – .i.e.:

$R^2=\dfrac{\sum_j (y_j – \bar{y})^2 – \sum_j (y_j – \hat{y})^2}{\sum_j (y_j – \bar{y})^2}$.

Using this method has resolved the problem I'm having when using the subset mean to calculate $R^2$, where I get a low or negative value if my subset has very low variance (e.g. if the subset is instances with target values in a small range), and I'd like to use Dave's method in a paper I'm writing.

I've searched for academic references for this method of calculating $R^2$ but I have not found one so far.

Does anyone know of a suitable reference to use for this?

Best Answer

As my question is so far unanswered and I haven't been able to find a reference, I'm posting how I resolved this as an answer.

In my paper I justified this approach with the statement "When $R^2$ is calculated for a subset of samples, $\bar{y}$ is the mean of the measured LFMC for the full sample set, not the mean of the subset. By using this value for the mean, all $R^2$ calculations are compared to the same baseline, thus allowing comparisons between $R^2$ for different subsets of samples."

Related Solutions

R – Calculating R-Squared with Centered vs. Uncentered Sums of Squares

As Stephane hinted in the comment, it is the difference between model with or without intercept that matters.

For what it is worth, here is code from one of my packages:

## cf src/library/stats/R/lm.R and case with no weights and an intercept
f <- object$fitted.values
r <- object$residuals
mss <- if (object$intercept) sum((f - mean(f))^2) else sum(f^2)
rss <- sum(r^2)
r.squared <- mss/(mss + rss)

Residuals are centered by design, leaves the fitted values with need to be centered in the intercept and not otherwise.

Solved – Computing c-index for an external validation of a Cox PH model with R

I just got an explanation from a colleague about how to use this feature. In the help page for rcorr.cens(), it states that x is a "numeric predictor variable". I thought that this meant it had to be a model variable like Age, Stage, Metastasis, etc. What I found out is that x can just be a vector of your model's survival estimates for an external data set. Therefore the only two things rcorr.cens() needs is that vector of survival estimates and a Surv() object. Using my code from above, this is how you use it:

library(rms)
surv.obj=with(veteran,Surv(time,status))   ####This will be used for rcorr.cens
cox.mod=cph(surv.obj~celltype+karno,data=veteran,x=T,y=T,surv=TRUE,time.inc=5*365)

##Here is the test data set that is the external "independent" data.
test_dat=data.frame(trt=replicate(500,NA), celltype=replicate(500,NA), time=replicate(500,NA), status=replicate(500,NA), karno=replicate(500,NA), diagtime=replicate(500,NA), age=replicate(500,NA), prior=replicate(500,NA))
for(i in seq(8)){
test_dat[,i]=sample(veteran[,i],500,replace=T)
}

###Create your survival estimates
estimates=survest(cox.mod,newdata=test_dat,times=5*365)$surv


###Determine concordance
rcorr.cens(x=estimates,S=surv.obj)

I hope this helps anyone in the future who has the same question!

Best Answer

Related Solutions

R – Calculating R-Squared with Centered vs. Uncentered Sums of Squares

Solved – Computing c-index for an external validation of a Cox PH model with R

Related Question