Background:
It's common in many aspects of science to want to determine if the trends shown in spatial data are significant. In my field of familiarity, climate science, authors often show the trends in temperature data, for example, and overlay stippling to show trend significance at 5%. For example see Rotstayn et al. (2013) (open access) Fig. 1:
Rotstayn et al. mention that
statistical significance of ensemble means is assessed using a two-sided t-test [Sect. 2]
For the example above, let's assume they have three data sets: model1 –> temperature climatology; model2 –> temperatures from an alternative model and model3 –> temperatures from another alternative model. (a) and (b) are then model2 – model1 and model3 – model1 respectively. In this case, the plot is latitude-pressure (height), but plots may be lat-lon, lon-pressure etc. Ultimately, you're working with a set of 2d matrices [x,y].
Question:
How do you obtain significance at each discrete data point rather than obtaining one t/p statistic for the comparison of the entire data set, such that you can plot of field of 5% significance.
I plot and process data in Python, so bonus points for discussion how this can be implemented in Python.
Also feel free to discuss the merits of t-tests and/or suggest alternative tests that may be better suited to data that is not normally distributed.
Note:
A similar question has been asked on Stackover flow here: Test for statistically significant difference between two arrays but I don't understand how the answers help me.
Best Answer
You need to adjust for multiple testing (which you are doing when you plot a map with stippling: multiple hypothesis tests — 1 test per grid cell). I.e. you can't stipple grid cells whose p-values are lower than the typically used 0.05 significance level. You need to calculate a new reduced significance level ("p_fdr"; eqn. 3 in the paper provided) by controlling for the False Discovery Rate and only stipple grid cells whose p-values lie under that value.
The has been ignored in the atmospheric sciences for decades now. A recent publication focuses on just this exact issue: “The Stippling Shows Statistically Significant Grid Points: How Research Results are Routinely Overstated and Overinterpreted, and What to Do about It"