Solved – Correcting for multiple 2-way anova testing

anovafalse-discovery-ratemultiple-comparisons

I want to study the effect of two factors on a paramater that I measure in several regions. I'm running a 2-way ANOVA test for each region (about a dozen) and I am wondering how I can deal to avoid the increase of the type I error. I'm thinking about performing an FDR correction on the p-values that I get with the ANOVA tests. I get 3 p-values for each test (one for each factor and the third for interaction between factors), so approximately 36 p-values altogether. I wonder how to appropriately correct for multiple tests and particularly whether I should correct the p-values all together or separately for each factor and for the interaction.

Thank you !

Best Answer

I don't know what exactly are the regions you are mentioning in your question but if your data are spatially correlated you should think of taking into account this correlation, by calculating a nonparametric statistical test.

In short, the algorithm could be described by the following steps:

  1. For every region, conduct the 2way ANOVA and compare your signals by means of an F-value.
  2. Select all regions whose F-value is larger than some threshold. (Usually this threshold corresponds to a p<.05. By changing it you affect the sensitivity/specificity of the test).
  3. Cluster the selected samples in connected sets on the basis of spatial adjacency.
  4. Calculate cluster-level statistics by taking the sum of the F-values within a cluster.

To construct your null distribution:

  1. Create a random partition of your data by permuting randomly the labels of the factors (that depends strongly on the design of your study ie., if you have within or between subjects factors or both).

  2. Calculate the test-statistic for this random partition by repeating steps 1-4.

  3. Repeat steps 5-6 for a fair amount of times (usually 10K) to construct a null distribution.

  4. Your p-value is the proportion of random partitions that resulted in a larger test statistic than the observed one.

You can check out these papers to get a detailed description of the implementation:

Holmes AP, Blair RC, Watson JDG, Ford I. Nonparametric analysis of statistic images from functional mapping experiments. J Cerebr Blood Flow Metabol 1996;16:7–22.

Bullmore E, Suckling J, Overmeyer S, Rabe-Hesketh S, Taylor E, Brammer M. Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain. IEEE Trans Med Imag 1999;18:32–42.

Maris E, Oostenveld R, Nonparametric statistical testing of EEG- and MEG-data, JNeuroscMethods, 2007; 164:177–190.

I can also point you out to a matlab-based free toolbox (I'm sure there are more but this is the one I know of) with bult-in functions to implement these kinds of tests: http://www.fieldtriptoolbox.org/.

Related Question