The p-values provided by the built-in Matlab and Pylab Pearson correlation functions are stated to be inaccurate for small sample sizes, or when the samples are not normally distributed. The python documentation suggests using an N>500 for the p-value to be accurate, while the Matlab documentation gives no specific cutoff other than "large".
Does anyone know a method that can correctly test for statistical significance of the Pearson correlation coefficient under such circumstances?
My instinct is that I could just perform a permutation test by scrambling the X or Y values and re-sampling the Pearson correlation coefficient from the scrambled data — then use that distribution to get a confidence interval. But, would this be correct?
Best Answer
In terms of the p-value, the answer can be found in an earlier post. Basically, use the permutation test for n<20. A generally normalizing transformation, such as rankit, will work for larger n's and will be more powerful (Bishara & Hittner, 2012). Of course, if you transform, you're no longer looking at the linear relationship on the original scale.
In terms of the confidence interval, the answer is less clear. There aren't many published large-scale Monte Carlo comparisons. Puth et al. (2014) have some evidence that the Fisher Z can be inadequate with large violations of normality. There was no general solution - even bootstrapping with BCa did not solve it. You might consider either:
a)Spearman CIs with Fisher Z. Instead of using $SE_z=1/\sqrt(n-3)$, use the Fieller et al. (1957) estimate of standard error for the Fisher Z: $SE_z=1.03/\sqrt(n-3)$
b)Transforming via rankit, and then using the Fisher Z for the CI as usual
References:
Bishara, A. J., & Hittner, J. B. (2012). Testing the significance of a correlation with non-normal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological Methods, 17, 399-417. doi:10.1037/a0028087
Fieller, E. C., Hartley, H. O., & Pearson, E. S. (1957). Tests for rank correlation coefficients. I. Biometrika, 44, 470-481.
Puth, M., Neuhäuser, M., & Ruxton, G. D. (2014). Effective use of Pearson’s product-moment correlation coefficient. Animal Behaviour, 93, 183-189.