Solved – Mahalanobis Distance on Singular Data

distancematrixr

I have an issue which I could not solve, although I tried and I got some help on R forum.

I am trying to calculate Mahalanobis distances on a data.frame, where I have several hundreds of groups and several hundreds of variables.
Whatever I do, I get the system is computationally singular: reciprocal condition number error.

It is clear that it is singular, but is there any way to get rid of it and run Mahalanobis? Should I forget solve this using another approach? If yes, then what else to use?

I have uploaded the data file to my FTP: It is a tab delimited txt file with no headers.

I was working with the R StatMatch Mahalanobis (also tried stats Mahalanobis) function. I have a deadline for this project (not a homework!), and I could always use this function, so I thought I will be able to keep the calculations short, but now I am lost.

Migrated from Stack Overflow.

Best Answer

Why do you think there is no way that matrix could be singular?

A QR decomposition shows that the rank of this 380 x 372 matrix is just 300. In other words, it is highly singular:

url <- "http://mkk.szie.hu/dep/talt/lv/CentInpDuplNoHeader.txt"
df <- read.table(file = url, header = FALSE)
m <- as.matrix(df)

dim(m)
# [1] 380 372
qr(m)$rank
# [1] 300

Examining the matrix's singular values is another way to see the same thing:

head(table(svd(df)$d))

# 5.76661502353373e-13 2.57650568058543e-12  0.00929562094651422 
#                   71                    1                    1 
#   0.0277990885015625   0.0398152894712022   0.0469713341003743 
#                    1                    1                    1

Related Solutions

Solved – Mahalanobis distance for vector-classification

Assuming there are some differences between the covariance matrices of the $X_i$ classified as $+1$ and those classified as $-1$, you could do the following:

Calculate the covariances for the two sets of $X_i$. I'll label them $\Sigma_{+}$ and $\Sigma_{-}$.
For all $i$ in the +-set: $d_{test,i} = \sqrt{(x_{test}-x_i)^{\text{T}}\Sigma_{+}^{-1}(x_{test}-x_i)}$. Similarly for the --set, just using $\Sigma_{-}^{-1}$ instead, obviously.
Take the $i$ associated with the minimum $d_{test,i}$ as your closest history-set vector.

The $d_{test,i}$ are the Mahalanobis distances between $X_{test}$ and the $X_i$.

Sample code in R for a single covariance matrix:

# Create sample history matrix with 100 entries 
X <- matrix(rnorm(1000),10,100)
# Create sample test matrix
Xtest <- rnorm(10)

# Calculate the Mahalanobis distances
Sigma <- cov(t(X))
SInv <- solve(Sigma)

di <- rep(0, ncol(X))
for (i in 1:length(di)) {
  di[i] <- sqrt(t(Xtest-X[,i]) %*% SInv %*% (Xtest-X[,i]))
}

which.min(di)

Solved – Why use the Mahalanobis distance

One of the issues is that all the variables could be on different scales. Suppose that two of your variables are income and gender, the former in dollars and the latter as a 0-1 indicator variable. Which is further away: being off by 1 unit in income or 1 unit in gender? Being just a dollar away is pretty good, but having the wrong sex is as far away as you can get. You need to normalize how far these distances are; standard Euclidean distance doesn't do this. The variance-covariance matrix rescales the variables to make distances more comparable.

Best Answer

Related Solutions

Solved – Mahalanobis distance for vector-classification

Solved – Why use the Mahalanobis distance

Related Question