correlationmachine learningscatterplot
If I have a large number of relationships I want to check that may or may not be linear (as in the bottom row of image below), is there a method that will reject the hypothesis the scatter plot is showing white noise? I want to check for the presence of any relationship without knowing what the form of that may be. What method/algorithm of measuring deviations from white noise captures the most different types of deviation?
image from:
https://en.wikipedia.org/wiki/Correlations
Your discussion is correct. Imagine a set of heights and weights measured with a lot of precision (this is made up data to illustrate a point -- it really won't be like a set of real heights and weights - but notionally these are cm and kg)
There are two common ways to look at such a plot (usually in slightly different situations, but sometimes just as different ways of looking at the same data).
The first is simply to consider the bivariate distribution of which the sample (in some circumstances) might represent a randomly drawn set of bivariate observations. Specifically, the points will be more dense in regions where the bivariate density is higher. You will be able to think about the association between the variables (such as the fact that height and weight tend to increase together).
The other way is to consider the distribution of the $y$-variable conditional on the $x$-variable. (Excuse me, I'm going to be a little loose with notation to avoid obscuring the central ideas.)
If you had many people at each height, you could consider the distribution at each height. If the distributions at different $x$'s are similar apart from the mean (say), then this view could be thought of as regarding the distribution in terms of the way the means move: $m(x) = E(y|x)$, and then the distribution about the mean at given $x$ values (the distribution of $e = y - m(x)$, which will have characteristics that don't depend on $x$).
Note that $m(x) = E(y|x)$ is of the functional form you're used to thinking about.
(More generally, of course, the distribution around isn't so nicely behaved and does vary with $x$, but in many situations it's a reasonable description.)
If you don't have many at each height, you can convey a similar concept by thinking in terms of not an exact $x$, but the distribution of $y$ in a narrow strip of $x$-values, to approximate the distribution in the center of the strip.
This concept can either be conceived over a set of non-overlapping strips, or in a window that is moving across the range of $x$ (where there is overlap).
(Where $E[y|x]$ was assumed to progress linearly, we would be talking about linear regression models, at least as a general concept.)
In this discussion I have slipped back and forth between sample and population concepts, and I have done things like talk about expectations while illustrating it with what is notionally a sample mean. Hopefully the distinction between sample and population quantities remains clear enough in spite of the way I've not tried to keep them apart.
If you look at your scatterplot again, you will see that this is not so much corners of a square as it is a kind of an X pattern. As Maartin Buis notes in the comments, this plot looks like it has two groups of data points, one of which has a strong positive relationship between the variables, and one of which has a slightly less strong negative relationship between the variables. If these groups are of roughly the same sizes (which they appear to be) then these correlations cancel out in the aggregate and so you are getting a result that shows low sample correlation.
In view of this, I would suggest you look for a third binary variable that might account for the apparent groups in this plot. You could try plotting your data again, but colouring them by any available binary variables, and see if this demarcates two groups with positive and negative correlation. If this is not available in your data then it might just be a case where you have a lurking variable that is causing mischief.
Best Answer
Maximal Information coefficient is one method that has been used for this. "In statistics, the maximal information coefficient (MIC) is a measure of the strength of the linear or non-linear association between two variables X and Y."
Detecting Novel Associations in Large Data Sets. D. Reshef et. al
http://en.wikipedia.org/wiki/Maximal_information_coefficient