Solved – Mixed Model Type-III Sums of Squares- R vs SPSS

mixed modelrspsssums-of-squares

The age old question of comparing sums of squares (SS) between programs has reared its ugly head again.

I am trying to replicate output in SPSS, that was computed using Type 3 Sums of Squares, in R.

I understand that with multiple regressions, there are several ways to get Type 3 SS in R (to match Type 3 output from SPSS).

However, I am running a mixed model using aov (which uses Type 1 SS) and even when I try all the "usual" fixes," my estimates don't match the Type 3 SS output from SPSS.

First of all, when I run the SPSS syntax using "/METHOD=SSTYPE(1)" the results match those I get using this code:

  mymodel<-aov(data=longdat,  DV ~ 1 + Task + Cue + Compatibility + Cue:Task + Compatibility:Task + Cue:Compatibility + Cue:Compatibility:Task + Error(subject/Cue/Compatibility/Cue*Compatibility))
  summary(mymodel)

So I know the analyses are the same when they use Type 1 SS.

However, when I use:

 options(contrasts = c("contr.sum","contr.poly"))
 tt<-lm(DV ~ 1 + Task + Cue + Compatibility + Cue:Task  + Compatibility:Task + Cue:Compatibility + Cue:Compatibility:Task + 1/subject/Cue/Compatibility    /(Cue*Compatibility), data=longdat)
 drop1(tt, ~., test="F")

The results do not match the SPSS Type 3 output.

In attempts to get matching output, I have also tried the Anova function (which can give Type 3 SS)

  Anova(mymodel, type=3, test.statistic="F")

but I get this error "Error in terms.formula(formula, data = data) : 'data' argument is of the wrong type."

I have also tried using lmer.

Can someone help me get Type 3 Sums of Squares for a mixed model in R?

Thank you!

Best Answer

I think the simplest thing to do will be to simply use the functions mixed and aov.car from the package afex. Please note that to calculate the degrees of freedom these functions use the Kenward-Rogers approximation. Other functions (or other software in general) might use Satterthwaite approximations. A very interesting discussion on Type-III sums was done in this site and is available: here.

Related Solutions

Type-III Sums of Squares – Should an Argument be Included to Request Type-III Sums of Squares in ezANOVA?

Just to amplify - I am the most recent requestor, I believe.

In specific comment on Mike's points:

It's clearly true that the I/II/III difference only applies with correlated predictors (of which unbalanced designs are the most common example, certainly in factorial ANOVA) - but this seems to me to be an argument that dismisses the analysis of the unbalanced situation (and hence any Type I/II/III debate). It may be imperfect, but that's the way things happen (and in many contexts the costs of further data collection outweigh the statistical problem, caveats notwithstanding).
This is completely fair and represents the meat of most of the "II versus III, favouring II" arguments I've come across. The best summary I've encountered is Langsrud (2003) "ANOVA for unbalanced data: Use Type II instead of Type III sums of squares", Statistics and Computing 13: 163-167 (I have a PDF if the original is hard to find). He argues (taking the two-factor case as the basic example) that if there's an interaction, there's an interaction, so consideration of main effects is usually meaningless (an obviously fair point) - and if there's no interaction, the Type II analysis of main effects is more powerful than the Type III (undoubtedly), so you should always go with Type II. I've seen other arguments (e.g. Venables, Fox) that emphasize the meaning (or lack of) of considering hypotheses about main effects in the presence of interactions, and/or/equivalently suggesting that the Type III assumptions about the null hypothesis are often not sensible (e.g. Langsrud).
And I agree with this: if you have an interaction but have some question about the main effect as well, then you're probably into do-it-yourself territory.

Clearly there are those who just want Type III because SPSS does it, or some other reference to statistical Higher Authority. I am not wholly against this view, if it comes down to a choice of a lot of people sticking with SPSS (which I have some things against, namely time, money, and licence expiry conditions) and Type III SS, or a lot of people shifting to R and Type III SS. However, this argument is clearly a lame one statistically.

However, the argument that I found rather more substantial in favour of Type III is that made independently by Myers & Well (2003, "Research Design and Statistical Analysis", pp. 323, 626-629) and Maxwell & Delaney (2004, "Designing Experiments and Analyzing Data: A Model Comparison Perspective", pp. 324-328, 332-335). That is as follows:

if there's an interaction, all methods give the same result for the interaction sum of squares
Type II assumes that there's no interaction for its test of main effects; type III doesn't
Some (e.g. Langsrud) argue that if the interaction is not significant, then you're justified in assuming that there isn't one, and looking at the (more powerful) Type II main effects
But if the test of the interaction is underpowered, yet there is an interaction, the interaction may come out "non-significant" yet still lead to a violation of the assumptions of the Type II main effects test, biasing those tests to be too liberal.
Myers & Well cite Appelbaum/Cramer as the primary proponents of the Type II approach, and go on [p323]: "... More conservative criteria for nonsignificance of the interaction could be used, such as requiring that the interaction not be significant at the .25 level, but there is insufficient understanding of the consequences of even this approach. As a general rule, Type II sums of sqaures should not be calculated unless there is strong a priori reason to assume no interaction effects, and a clearly nonsignificant interaction sum of squares." They cite [p629] Overall, Lee & Hornick 1981 as a demonstration that interactions that do not approach significance can bias tests of main effects. Maxwell & Delaney [p334] advocate the Type II approach if the population interaction is zero, for power, and the Type III approach if it isn't [for the interpretability of means derived from this approach]. They too advocate using Type III in the real-life situation (when you're making inferences about the presence of the interaction from the data) because of the problem of making a type 2 [underpowered] error in the interaction test and thus accidentally violating the assumptions of the Type II SS approach; they then make similar further points to Myers & Well, and note the long debate on this issue!

So my interpretation (and I'm no expert!) is that there's plenty of Higher Statistical Authority on both sides of the argument; that the usual arguments put forward aren't about the usual situation that would give rise to problems (that situation being the common one of interpreting main effects with a non-significant interaction); and that there are fair reasons to be concerned about the Type II approach in that situation (and it comes down to a power versus potential over-liberalism thing).

For me, that's enough to wish for the Type III option in ezANOVA, as well as Type II, because (for my money) it's a superb interface to R's ANOVA systems. R is some way from being easy to use for novices, in my view, and the "ez" package, with ezANOVA and the rather lovely effect plotting functions, goes a long way towards making R accessible to a more general research audience. Some of my thoughts-in-progress (and a nasty hack for ezANOVA) are at http://www.psychol.cam.ac.uk/statistics/R/anova.html .

Would be interested to hear everyone's thoughts!

Solved – Type III sums of squares

I've found differences in the estimation of regressors between R 2.15.1 and SAS 9.2, but after updating R to 3.0.1 version the results were the same. So, first I suggets you to update R to the latest version.

You're using the wrong approach because you're calculating the sum of square against two different models, which implies two different design matrices. This lead you to totally different estimation in the regressors used by lm() to compute the predicted values (you're using regressors with different values between the two models). SS3 is computed based on an hypotesis test, assuming that all the conditioning regressors equal zero, while the conditioned regressor equals 1. For the computations, you use the same design matrix used to estimate the full model, as for the regressor estimated in the full model. Remember that the SS3s aren't full additive. This means that if you sum the estimated SS3, you don't obtain the model SS (SSM).

Here I suggest a R implementation of the mathematics that implements the GLS algorithm used to estimate SS3 and regressors.

The values generated by this code are exactly the same generated using SAS 9.2 as for the results you gave in your code, while the SS3(B|A,AB) is 0.167486 instead of 0.15075. For this reason I suggest again to update your R version to the latest available.

Hope this helps :)

A<-as.factor(rep(c("male","female"), each=5))
set.seed(1)
B<-runif(10)
set.seed(5)
y<-runif(10)


# Create a dummy vector of 0s and 1s
dummy <- as.numeric(A=="male")

# Create the design matrix
R <- cbind(rep(1, length(y)), dummy, B, dummy*B)

# Estimate the regressors
bhat <- solve(t(R) %*% R) %*% t(R) %*% y
yhat <- R %*% bhat
ehat <- y - yhat

# Sum of Squares Total
# SST <- t(y)%*%y - length(y)*mean(y)**2
# Sum of Squares Error
# SSE <- t(ehat) %*% ehat
# Sum of Squares Model
# SSM <- SST - SSE

# used for ginv()
library(MASS)

# Returns the Sum of Squares of the hypotesis test contained in the C matrix
SSH_estimate <- function(C)
{
    teta <- C%*%bhat
    M <- C %*% ginv(t(R)%*%R) %*% t(C)
    SSH <- t(teta) %*% ginv(M) %*% teta
    SSH
}

# SS(A|B,AB)
# 0.001481682
SSH_estimate(matrix(c(0, 1, 0, 0), nrow=1, ncol=4))
# SS(B|A,AB)
# 0.167486
SSH_estimate(matrix(c(0, 0, 1, 0), nrow=1, ncol=4))
# SS(AB|A,B)
# 0.01627824
SSH_estimate(matrix(c(0, 0, 0, 1), nrow=1, ncol=4))

Best Answer

Related Solutions

Type-III Sums of Squares – Should an Argument be Included to Request Type-III Sums of Squares in ezANOVA?

Solved – Type III sums of squares

Related Question