Solved – Lack of fit test in multiple regression

multiple regressionregressionself-study

Suppose I have this model
$$Y=B_0+B_1X_1+B_2X_2$$
and these observations

Y  <- c(64, 73, 61, 76, 72, 80, 71, 83, 83, 89, 86, 93, 88, 95, 94, 100)
X1 <- c(4, 4, 4, 4, 6, 6, 6, 6, 8, 8, 8, 8, 10, 10, 10, 10)
X2 <- c(2, 4, 2, 4, 2, 4, 2, 4, 2, 4, 2, 4, 2, 4, 2, 4)

I know how to calculate SSLF (1) and SSPE (2) in R, but I want to know how to do it at hand
$$SSLF=SSE-SSPE$$
where
$$SSE=\sum(Y_i-\hat{Y_i})^2$$

setting up the table this way is easy to calculate the sum of squares of pure error, but there is an easier way to do this?

(1) SSLF: sum of squares of lack of fit

(2) SSPE: sum of squares of pure error

Best Answer

Let me give you a hint. The SSPE is made up of squared deviations from the means at each $X$ level. Let's denote the number of $X$ levels by $c$, then

$$SSPE=\sum_j^c \sum_{i}^{n_j} \left(Y_{ij} - \bar{Y_j} \right)^2$$.

Simply put, for a replicate, i.e. an identical $X$ value, you compute the mean of the the corresponding $Y$s and sum the squared deviations from it. It is easy to see that any $X$ level with no replications makes no contribution to SSPE because the mean is just that one observation!

Hope this clears it up a bit.

Related Solutions

Solved – DF for regression SS in multiple linear regression when sigma is known

There is a common confusion in your understanding of the degrees of freedom in the typical situation. The degrees of freedom for a multiple regression is often written $\text{df}=N-p-1$. I think this is where you are getting the idea that the df for the regression is $p-1$. I typically prefer to write this as $\text{df}=N-(p+1)$ for this reason. Note that $-1*(p+1)=(-p-1)$, hence: $N-(p+1)=N-p-1$. This is a small piece of algebra that I think is slightly less intuitive for people.

Let's work though the degrees of freedom and see if we can make more sense of it:

If you simply have $N$ data, and are doing nothing with them, then I suppose you could say you have $N~\text{df}$.
If you wanted to calculate the mean of your data, then you have used $1~\text{df}$, so now you have only $N-1~\text{df}$ left. The standard way of explaining this is that all but one of your data could be any old real number, but in order to end up with the mean you've found, the last datum will be forced to be some particular value--it wouldn't be 'free to vary'.
So, what if you wanted to fit a simple (i.e., 1 $X$ variable) regression model? Then you are estimating the mean of $Y$, i.e. $\bar y$, and the slope of the relationship between $X$ and $Y$, $\hat\beta_1$. (Most people typically think of this as estimating the slope, $\hat\beta_1$, and the intercept, $\hat\beta_0$, but note that once you have $(\bar x,~\bar y)$ and $\hat\beta_1$, the intercept falls out of this: $\hat\beta_0=\hat\beta_1\bar x-\bar y$; and remember that $\bar x$ is free / doesn't cost any degrees of freedom, because your $X$ variable is assumed to be a set of known constants.)
And what if you wanted to fit a multiple regression model with $p$ $X$ variables? Then you would be estimating the mean of $Y$ and $p$ slopes, i.e., $\bar y$ and $\hat\beta_j$ (or $p$ slopes, $\hat\beta_j$, and the intercept, $\hat\beta_0$, if that's more intuitive for you). Thus, you would have $N-(p+1)~\text{df}$ remaining.
Importantly, recognize that nowhere in this discussion have we discussed estimating $\hat\sigma^2_\varepsilon$ yet, nor is that part of calculating the $\text{df}$ for our model.

The residual variance has to do with how we form intervals. For example, if you wanted to form a confidence interval around a slope, $\hat\beta_j$, to test whether it was equal to some null value (typically $0$). If you know the residual variance a-priori, and your residuals are normal (or, your sample is sufficiently large for the Central Limit Theorem to cover for you), then you can use the normal distribution to form this CI / to test your slope. On the other hand, if you estimated your residual variance from your data, then you would use the $t$ distribution for this, with the model's $\text{df}$.

(Response to edit:)

I don't quite follow the thinking behind your question; I continue to suspect there is a misunderstanding at play. The ANOVA table for a regression model, e.g., is constructed like this:

\begin{array}{lllll} &\text{Source} &\text{SS} &\text{df} &\text{MS} &\text{F} \\ \hline &\text{Regression} &\sum(\hat y-\bar y)^2 &p &\frac{\text{SS}_{reg}}{\text{df}_{reg}} &\frac{\text{MS}_{reg}}{\text{MS}_{res}} \\ &\text{Residual} &\sum(y_i-\hat y)^2 &N-(p+1) &\frac{\text{SS}_{res}}{\text{df}_{res}} \\ &\text{Total} &\sum(y_i-\bar y)^2 &N-1 \end{array}

To be explicit:

The sum of squares due to regression has $p\text{ df}$, not $p-1$.
Knowledge of $\sigma^2_\varepsilon$ (or the lack thereof) is unrelated to the $\text{df}$.
The parameters aren't free to vary, the data are.

Solved – Why is $SST=SSE + SSR$? (One variable linear regression)

Adding and subtracting gives \begin{eqnarray*} \sum_{i=1}^n (y_i-\bar y)^2&=&\sum_{i=1}^n (y_i-\hat y_i+\hat y_i-\bar y)^2\\ &=&\sum_{i=1}^n (y_i-\hat y_i)^2+2\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)+\sum_{i=1}^n(\hat y_i-\bar y)^2 \end{eqnarray*} So we need to show that $\sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=0$. Write $$ \sum_{i=1}^n(y_i-\hat y_i)(\hat y_i-\bar y)=\sum_{i=1}^n(y_i-\hat y_i)\hat y_i-\bar y\sum_{i=1}^n(y_i-\hat y_i) $$ So, (a) the residuals $e_i=y_i-\hat y_i$ need to be orthogonal to the fitted values, $\sum_{i=1}^n(y_i-\hat y_i)\hat y_i=0$, and (b) the sum of the fitted values needs to be equal to the sum of the dependent variable, $\sum_{i=1}^ny_i=\sum_{i=1}^n\hat y_i$.

Actually, I think (a) is easier to show in matrix notation for general multiple regression of which the single variable case is a special case: \begin{eqnarray*} e'X\hat\beta &=&(y-X\hat\beta)'X\hat\beta\\ &=&(y-X(X'X)^{-1}X'y)'X\hat\beta\\ &=&y'(X-X(X'X)^{-1}X'X)\hat\beta\\ &=&y'(X-X)\hat\beta=0 \end{eqnarray*} As for (b), the derivative of the OLS criterion function with respect to the constant (so you need one in the regression for this to be true!), aka the normal equation, is $$ \frac{\partial SSR}{\partial\hat\alpha}=-2\sum_i(y_i-\hat\alpha-\hat\beta x_i)=0,$$ which can be rearranged to $$ \sum_i y_i=n\hat\alpha+\hat\beta\sum_ix_i $$ The right hand side of this equation evidently also is $\sum_{i=1}^n\hat y_i$, as $\hat y_i=\hat\alpha+\hat\beta x_i$.

Best Answer

Related Solutions

Solved – DF for regression SS in multiple linear regression when sigma is known

Solved – Why is $SST=SSE + SSR$? (One variable linear regression)

Related Question