Solved – Simple main effects and separate t-tests

anovat-test

Lets say I have 2 factors and fully balanced groups:

  1. Between-subjects: condition (0, 1)
  2. Within-subjects: time (time1, time2)

I run a 2×2 between-within ANOVA and find a significant interaction. I proceed to test the simple main effects. I'm primarily interested in whether condition0 and condition1 are different from each other at time2

Is that simple main effect test the same as if I just used the time2 data and ran an independent samples t-test between the two conditions?

For reference, these are the two tests I'm referring to in SPSS syntax:

GLM time1 time2 BY condition
  /WSFACTOR=time 2 Polynomial 
  /METHOD=SSTYPE(3)
  /EMMEANS=TABLES(time*condition) compare(condition)
  /CRITERIA=ALPHA(.05)
  /WSDESIGN=time 
  /DESIGN=condition.

T-TEST GROUPS=condition(0 1)
  /MISSING=ANALYSIS
  /VARIABLES=time2
  /CRITERIA=CI(.95).

I was always of the understanding that the tests are different and should give different results, and the answer here implies that simple main effect tests are not the same as separate t-tests. However, I created some normally distributed test data, ran the code above, and all of my results are identical

Should I expect to always see the same results (p value, degrees of freedom, standard error, CI values etc)? More importantly, under what conditions would you expect the two sets of results to differ?

If they are always expected to be the same, why would you ever run the (arguably) more complicated simple main effects test and not just use t-tests every time (assuming an interaction has been found)?

Best Answer

They are not equivalent, and the reason boils down to the error term. When you conduct a t-test, you compute your error term based only on the subset of data being compared. However when you conduct the ANOVA, you’re using a different error term taking all data into account. From a strict statistical standpoint, if you can model the entire dataset, you should, and that error term should be used.

Related Question