I have two test groups that conducted an online task measuring response times (avg, avg(congruent), avg(incongruent)). I expected one group to be faster than the other but it turned out exactly the opposite way (significant). So now I'm trying to find one or more factors that are responsible for this unexpected outcome.
My test data looks like this:
participants conducted a Simon task and I measured their response times for each trial. In SPSS I have 3 variables, one for their average response times over all trials, one with their average response time for the trials that were congruent (place on screen and direction match), one with their average response time for the trials that were incongruent (place on screen and direction do not match).
My groups are monolingual and multilingual.
Literature suggests that multilinguals would score either higher averages overall, or higher averages for just the incongruent trials. My outcomes show that the monolinguals have a higher overall average and that the difference is the same between congruent and incongruent trials for both groups.
For each group I know the following variables:
- age
- education level
- accuracy%
- sex
- lurking variables (yes|no)
- average response times (total, congruent trials, incongruent trials)
- first language
Each variable in itself seems to be significant, but none of the variables seem to have a significant influence in combination with the original groups.
I want to ensure that my test was not faulty and verify the validity of the test method. Maybe one other factor I didn't account for in my hypothesis is causing these results. It would either help me substantiate that my test is correct and that the results are valid, or provide a new angle for future research.
I'm not sure what test to use. For my first comparisons I used Spearman's correlation. I was told once that I may not use partial correlations when using Spearman's correlation.
Can anyone help me how to proceed from here?
Best Answer
You can probably look at your data in another way. Start with a t-test of the difference in average response time between the groups (that would be equivalent to a test that the Pearson – not Spearman – correlation between group membership and average time is 0). From there, you can easily add other variables in the model (i.e. turn it into an ANOVA or linear regression), consider transformations, rank-based statistics or generalized linear model if needed, etc.
Because of the specifics of this type of experiments, it's also standard practice to analyze only successful trials and to exclude large response times before computing the means (there are better and more principled ways to deal with this problem but you definitely need to do something about it).
It could also be more appropriate to analyze the response time to individual trials directly, using a multilevel model (see the literature on the “language-as-fixed-effect fallacy” in psycholinguistics).
@AdamO is right, thought, doing all this after the fact is at best suggestive. If you fiddle with the model until you get something you like, p-values become meaningless. Also, you might have heard of the mounting debate on reproducibility within psychology. I personally think that the ease with which we explain away unexpected results through ancillary variables or details of the procedure is part of the problem. The effect might not be what you expected but a second look at the literature might reveal that it was not as strong as it first seems. If that's the case, the “disappearance” of the effect really does not need any explanation.