MATLAB: Getting a NaN in correlation coefficient

corrcorr2correlation coefficientmissingnannan's

Hi, i have a simple problem which unfortunately i am unable to understand.
I have matrices and i am trying to calculate correlation coefficient between two variables. A simple example from my code is attched. Why am i getting a NaN here. What does this implies
x=[-7.501899598769999514e-04;-6.501899598769999514e-04;-5.501899598769999514e-04];
y=[-0.414;-0.414;-0.414];
c11=corr2(x,y)

Best Answer

When NaNs appear in the output but are not present in the inputs
Notice that all of the values in y are identical y=[-0.414; -0.414; -0.414];
If you look at the equations for corr2() or Pearson's corr() you'll notice that both have a term in the denominator that subtracts the mean of y from each y-value. When each value of y is identical, the result is a vector of 0s. When you divide by zero, you get NaN.
Another way of putting it, the standard deviation of x or y cannot be 0. When you have a vector of identical values, the std is 0.
The NaN, in this case, is interpretted as no correlation between the two variables. The correlation describes how much one variable changes as the other variable changes. That requires both variables to change.
NaN values in the inputs spreading to the outputs
For r=corr2(x,y):
When there is 1 or more NaN values in the inputs, to corr2(x,y), the output will be NaN. Fill in the missing data before computing the 2D correlation coefficient.
For r=corr(x):
A single NaN value in position (i,j) of input matrix x will result in a full row of NaN values at row i and a full column of NaN values in column j of the output matrix r (see explanation). 
x = [
     6     5     1
     3   NaN     9
     5     3     7
     9     5     5 ];
 
 r = corr(x)
            1          NaN     -0.52699
          NaN          NaN          NaN
     -0.52699          NaN            1
For r=corr(x,y):
A single NaN value in position (i,j) of either x or y inputs will results in a column of NaN values in column j of the output matrix r. 
x = [
     9     5     1
     1     4     4
     2     6     4
     2     5     9 ];
y = [
     6     5     1
     3   NaN     9   
     5     3     7
     9     5     5 ];
 
 r = corr(x,y)
       0.1623          NaN     -0.92394
       0.3266          NaN     -0.23905
      0.62312          NaN      0.32367
Ignoring NaNs in corr() inputs
The rows option in corr() can be set to complete or pairwise which will ignore NaN values using different methods.
'rows','complete' removes the entire row if the row contains a NaN. In other words, it will remove row 2 from both x and y input matrices. Using the same inputs above,
r = corr(x,y,'rows','complete')
     -0.27735          0.5     -0.94491
     -0.69338           -1      0.75593
      0.81224      0.14286      0.53995
      
      
r2 = corr(x,y) % for comparison

       0.1623          NaN     -0.92394
       0.3266          NaN     -0.23905
      0.62312          NaN      0.32367
Notice that this changes all of the correlation values since the entire row #2 was removed from both inputs x and y. To confirm that, we can remove those rows and recompute the correlation matrix. 
% Remove row 2 which contains a NaN in y

r3 = corr(x([1,3,4],:) ,y([1,3,4],:));  
     -0.27735          0.5     -0.94491
     -0.69338           -1      0.75593
      0.81224      0.14286      0.53995
Voila! Outputs r and r3 match.
'rows','pairwise' only removes rows only if a NaN appears in the pairing of two columns. For the same x, y inputs as above, the correlation with columns in x paired with the 2nd column in y will omit the NaN and will be based on the remaining 3 values. All other column-paired correlations will use all 4 rows of values. 
r = corr(x,y,'rows','pairwise')
       0.1623          0.5     -0.92394
       0.3266           -1     -0.23905
      0.62312      0.14286      0.32367
      
      
r2 = corr(x,y) % for comparison
       0.1623          NaN     -0.92394
       0.3266          NaN     -0.23905
      0.62312          NaN      0.32367
Notice that values in columns 1 and 3 haven't changed since they do not involve column #2 in y. To confirm the correlation values in column 2 of r, 
% Remove row 2 which contains a NaN in y
r3 = corr(x([1,3,4],:) ,y([1,3,4],:)); 
% Replace NaN column in r2 with new r values
r2(:,2) = r3(:,2)
       0.1623          0.5     -0.92394
       0.3266           -1     -0.23905
      0.62312      0.14286      0.32367
Voila! Updated output r2 matches r.
Related Question