Solved – Permutation tests with repeated measures

permutation-testrepeated measures

Although reading quite a bunch of books, I'm still not sure, which method to use and how to implement it, therefore I appreciate any help!

I have 4 different groups (treatments) with 50 participants each. Each participant's action is observed 5 times under the same condition. The 5 different values are collected within one "round", where each participant has to select 5 items at an arbitrary time.
So I know that the different values for each participant are not independent of each other. However, time effects are not of interest for me and I don't assume that learning effects are apparent. I just collect 5 values for each participant in order to get more data. All the data within one group (for all users and their choices) is then taken together and a complex aggregated measure (a number) is calculated for all groups. Now the question is, whether there are significant differences for this measure between the groups. As there is no unbiased estimator for the measure and as my data is not normal, I know that I need some form of sampling. So the Welch or Levene tests, for instance, would not work here.

I'm just not sure whether bootstrapping or permutation tests seem to be more appropriate.
Second, could I "ignore" the repeated measures and permute all values together without regard to the participants who provided it? If not, am I supposed to create vectors for each participant and permute these?
What happens if I have an unbalanced design (in case some observations are missing)?
Do you know any software where such a method is already implemented?

Best Answer

If you permute the individual values then you are testing the combined null hypothesis that there is no difference between groups and that there is no structure within values from the same individual. So if you reject the null you don't know if it is because the groups differ or because there is structure within an individual.

I would suggest permuting individuals and keeping all values with the individual.

I don't know of a program that does this out of the box, but writing a permutation test in R is often easier/quicker than figuring out a precanned program.

Here is a simple example:

library(nlme)

Odist <- split(Orthodont$distance, Orthodont$Subject)

out <- replicate( 1999, { tmp <- sample(Odist);
    mean( unlist( tmp[1:16] ) ) - mean( unlist( tmp[17:27] ) ) } )

out <- c(out, mean(unlist(Odist[1:16])) - mean(unlist(Odist[17:27])) )

hist(out)
abline( v=out[2000] )

mean(out >= out[2000])
Related Question