Solved – Correlation analysis and correcting $p$-values for multiple testing

correlationmultiple-comparisonspermutation-test

I am investigating the effect of a drug on EEG and cognition in epilepsy patients.
We have done EEG and neuropsychological (NP) tests twice, before and after medical treatment.
Some of parameters in EEG and NP tests were significantly altered after drug treatment (done by Wilcoxon rank sum test, as the distribution was not normal).

Now, I would like to calculate correlation between change in EEG parameters and change in NP tests.
There are 7 parameters in EEG parameters and 27 in NP tests. I would like to run correlation analysis on every pair and see if any pair is significantly correlated.

My questions are:

  1. Should I use nonparametric correlation, say Spearman, to calculate $\rho$? (as values of EEG and NP paramaters are not normal)

  2. Since there are 7 parameters in EEG and 27 parameters in NP tests, there should be 7*27 number of testing, which requires correction of p value for multiple comparisons. Which method should I use?

I am looking for not-so-conservative method. I have seen from another posting that I can use permutation test for multiple comparisons. Could someone explain how I can do this?

Best Answer

In my opinion you should test if your variables are distributed normally and chose a suitable test accordingly.

Concerning the correction for alpha inflation: What you are doing is data mining. You have experimental data and now you are digging in it to find ... anything. Do that. But also know that anything you might find is just an observation and as such not reliable. Perform that exploratory thing, pick some promising pairs of variables and conduct new experiments to test these pairs for correlation.

Related Question