Mantel test and Moran's I refer to two very different concepts.
The reason for using Moran's I is the question of spatial autocorrelation: correlation of a variable with itself through space. One uses Moran's I when wants to know to which extent the occurrence of an event in an areal unit makes more likely or unlikely the occurrence of an event in a neighboring areal unit. In other words (using your example): if there is a noisy crow on a tree, how likely or unlikely are there other noisy crows in the neighborhood? The null hypothesis for Moran's I is no spatial autocorrelation in the variable of interest.
The reason for using the Mantel test is the question of similarities or dissimilarities between variables. One uses the Mantel test when wants to know whether samples that are similar in terms of the predictor (space) variables also tend to be similar in terms of the dependent (species) variable. To put it simply: Are samples that are close together also compositionally similar and are samples that are spatially distant from each other also compositionally dissimilar? Using your example: it tests whether quiet crows are located near other quiet crows, while noisy crows have noisy neighbors. The null hypothesis is no relationship between spatial location and the DV.
Besides this, the partial Mantel test allows comparing two variables while controlling for a third one.
For example, one needs the Mantel test when compares
- Two groups of organisms, which form the same set of sample units;
- Community structure before and after disturbance;
- Genetic/ecological distance and geographic distance.
Here is a good discussion on the Mantel test and its application.
(Edited in response to Ladislav Nado's new examples)
If I may guess, the reason for your confusion is that you keep thinking of space and noise in your examples either as of two continuous variables, or as of one distance matrix (position in space) and one continuous variable (noise). In fact, to analyze similarities between two such variables, one should think of both of them as distance matrices. That is:
- one matrix (for example, for space) describes the differences for each pair of geographic coordinates. Value for 2 crows sitting next to each other is lower than the value for crows sitting far apart;
- another matrix (for environmental, genetic, or any other structure) describes the differences between measured outcomes at given points. The value for 2 crows with a similar level of noise (it doesn't matter if they are quiet or noisy--it's just a measure of similarity!) is lower than the value for a pair of crows with dissimilar levels of noise.
Then the Mantel test computes the cross-product of the corresponding values in these two matrices. Let me underline again that the Mantel statistic is the correlation between two distance matrices and is not equivalent to the correlation between the variables, used to form those matrices.
Now let's take two structures you showed in pictures A and B.
In picture A, the distance in each pair of crows corresponds to similarities in their level of noise. Crows with small differences in their level of noise (each quiet crow vs. another quiet crow, each noisy crow vs. another noisy crow) stay close, while each and every pair of crows with big difference in their level of noise (a quiet crow vs. a noisy crow) stay away from each other. The Mantel test correctly shows that there is a spatial correlation between the two matrices.
In picture B, however, the distance between crows does not correspond to the similarities in their level of noise. While all noisy crows stay together, quiet crows may or may not stay close. In fact, the distance in some pairs of dissimilar crows (one quiet+one noisy) is smaller than the distance for some pairs of similar crows (when both are quiet).
There is no evidence in picture B that if a researcher picks up two similar crows at random, they would be neighbors. There is no evidence that if a researcher picks up two neighboring (or not so distant) crows at random, they would be similar. Hence, the initial claim that On both plots the hypothesis valid is incorrect. The structure as in picture B shows no spatial correlation between the two matrices and accordingly fails the Mantel test.
Of course, different types of structures (with one or more clusters of similar objects or without clear cluster borders at all) exist in reality. And the Mantel test is perfectly applicable and very useful for testing what it tests. If I may recommend another good reading, this article uses real data and discusses Moran's I, Geary's c, and the Mantel test in quite simple and understandable terms.
Hope everything is slightly more clear now; though, I can expand this explanation if you feel like there is still something missing.
Best Answer
First, let's clear up some terminology: the term "data frame" doesn't refer to anything mathematical. It's a data structure in R and Pandas for working with tabular data. For statistical questions, I think it's better to think of data frames and matrices as interchangeable.
Second, Pearson correlation is defined on vectors. You're probably thinking of a correlation matrix, which is a matrix of pairwise correlations between more than two vectors, such as the vectors representing different variables in a tabular data set (which you could in turn represent in R as a data frame). The Pearson correlation test is likewise defined on vectors.
Meanwhile the Mantel test is defined on true matrices. However it is actually intended for use specifically on distance matrices, which are derived from data vectors. So both the Pearson correlation test and Mantel test can be used to test relationships between vectors. The difference lies in the question that each test is designed to answer. The Pearson correlation test answers something like the question
while the Mantel test answers something like the question
There's a very nice explanation of Mantel's test here.
To answer the second part of your question: Pearson's correlation coefficient is used "under the hood" in both tests. It's just used differently.