Solved – Three-way mixed design ANOVA or two two-way mixed design ANOVAs

manovapost-hocspss

I am desperately hoping for your quick help…
Would you please cross-validate 😉 my statistical proceedure and help me solve a huge problem (see c)??

So imagine there is an experiment to test a treatment and a control group. It consists of

a) behavioural tests (Test 1, Test 2)

b) questionnaires (metric scores were derived, so Q1, Q2, Q3…)

c) EEG-experiment with a test with two conditions within it (A and B)

The question is, whether the groups differ, and if, where exactly.

So to test this

for a)
I can use for each test (1 and 2) either independent t tests (for normally distributed values) or Mann-Whitney U test (for not normally distributed values), with p-values corrected (Sidak)

OR
should I use two-way ANOVAs for each test (values as within, group as between factor)?

for b)
same as a) ?

for c)
If I was to concider 2 electrodes (since I do not know where exactly to expect the effects) and have 2 conditions (A and B) in two groups (treated vs control) do I choose
the three-way mixed design ANOVA (2x2x2)?

OR
may I concider each electrode seperately to run a two-way mixed design ANOVA
with condition as within and group as between factors. Or would that yield an alpha-error accumulation?

May I do post hoc tests if there is no three-way interaction, but only two-way?
And when I do any of these, SPSS refuses to do post-hoc tests ("there are not more than two groups"), so do I use independent t tests for any significant interactions?

I am aware that it is a lot, but hope you might be so kind to help me!!

Thank you in advance

Best Answer

I think the choice to use a full factorial or separate tests depends mostly on what is common practice in your field. But I can also give some recommendations:

A and B: I don't think it would be a good idea to run an ANOVA with different tests or questionnaires unless they ask the same thing with a different purpose. If they test for different things, then they should be analysed separately. If they test for the same thing, but induce a pseudo-condition by the way they are conducted, then you could justify a full-factorial design.

C: A very reasonable approach would be to analyse each electrode separately as a 2-way ANOVA. You could even adjust the hypotheses for the fact that you ran two tests "not knowing where the effect would be." However, if the two placements were used based on some hypothesis, then you could justify not adjusting. However, if you do want to test the 3 way interaction (does the treatment have a different effect depending upon the condition and does this vary with electrode) then you must do the full 3-way analysis.

I don't know how SPSS works, but if you have a significant 2-way interaction, then you should be able to do "simple effects" testing to determine what condition or electrode the treatment effect differed.

Without an interaction and only a main effects model, then no, you don't need post-hoc tests because there are only two levels, so it is testing for the difference in means between the two levels of each factor.

Finally, it should also be said that if you did not fully randomise when your levels were given(i.e. all of test 1 or all of Q1 or condition A was tested first) then you cannot put all of them into one ANOVA anyway because the different effects between tests/Qs/conditions would be confounded with time/learning of the procedure.

Related Solutions

Solved – Post hoc test in a 2×3 mixed design ANOVA using SPSS

Answer edited to implement encouraging and constructive comment by @Ferdi

I would like to:

provide an answer with a full contained script
mention one can also test more general custom contrasts using the /TEST command
argue this is necessary in some cases (ie the EMMEANS COMPARE combination is not enough)

I assume to have a database with columns: depV, Group, F1, F2. I implement a 2x2x2 mixed design ANOVA where depV is the dependent variable, F1 and F2 are within subject factors and Group is a between subject factor. I further assume the F test has revealed that the interaction Group*F2 is significant. I therefore need to use post hoc t-tests to understand what drives the interaction.

MIXED depV BY Group F1 F2 
  /FIXED=Group F1 F2 Group*F1 Group*F2 F1*F2 Group*F1*F2 |  SSTYPE(3) 
  /METHOD=REML 
  /RANDOM=INTERCEPT | SUBJECT(Subject) COVTYPE(VC) 
  /EMMEANS=TABLES(Group*F2) COMPARE(Group) ADJ(Bonferroni)
  /TEST(0) = 'depV(F2=1)-depV(F2=0) differs between groups' 
    Group*F2 1/4 -1/4 -1/4 1/4 
    Group*F1*F2 1/8 -1/8 1/8 -1/8 -1/8 1/8 -1/8 1/8 
  /TEST(0) = 'depV(Group1, F2=1)-depV(Group2, F2=1)' Group 1 -1
    Group*F1 1/2 1/2 -1/2 -1/2 
    Group*F2 1 0 -1 0  
    Group*F1*F2 1/2 0 1/2 0 -1/2 0 -1/2 0 .

In particular the second t-test corresponds to the one performed by the EMMEANS command. The EMMEANS comparison could reveal for example that depV was bigger in Group 1 on the condition F2=1.

However the interaction could also be driven by something else, which is verified by the first test: the difference depV(F2=1)-depV(F2=0) differs between groups, and this is a contrast you cannot verify with the EMMEANS command (at least I did not find an easy way).

Now, in models with many factors it is a bit tricky to write down the /TEST line, the sequence of 1/2, 1/4 etc, called L matrix. Typically if you get the error message: "the L matrix is not estimable", you are forgetting some elements. One link that explains the receipt is this one: https://stats.idre.ucla.edu/spss/faq/how-can-i-test-contrasts-and-interaction-contrasts-in-a-mixed-model/

ANOVA – Post-Hoc for 2×2 Mixed Design ANOVA Using SPSS

There's no use for a post hoc test here. What could you possibly find in a post hoc that isn't obvious from the ANOVA? There's a main effect of A and a main effect of B and an interaction.

Some people do them to do something like test whether A1B1 - A1B2 is significant while A2B1 - A2B2 is not. They find that result and report it as important but it's meaningless because it tells you less than the interaction already told you. Significant and not-significant is not a test of differences between conditions dependent upon the level of the other condition. The interaction already was that test.

Best Answer

Related Solutions

Solved – Post hoc test in a 2×3 mixed design ANOVA using SPSS

ANOVA – Post-Hoc for 2×2 Mixed Design ANOVA Using SPSS

Related Question