Let's say we are in an $N$-dimensional space and that we have a large set of data. The distribution of this $N$-dimensional point cloud can be modeled by a multivariate Gaussian mixture model (estimated using the EM algorithm).
Now, given two data points $x$ and $y$ from my dataset, which distance can-I use to evaluate the closeness of these two points given the global distribution ?
In the case of a single Gaussian, I think that the Mahalanobis distance would be a natural choice, but when the global distribution is known to be non-Gaussian and a GMM is used to model the data, we end-up with $K$ different covariance matrices (one for each Gaussian).
Should I compute $K$ Mahlanobis distances (one for each covariance matrix) then do a weighted sum using the posterior probabilities over the Gaussians ? Something like this maybe :
\begin{equation}
dist(x,y)=\sum_{i=1}^K \frac{\gamma_i(x)+\gamma_i(y)}{2} \sqrt{(x-y)^T\Sigma_i^{-1}(x-y)}
\end{equation}
where $\Sigma_i$ is the covariance matrix of the $i$-th Gaussian and $\gamma_i(x)$ is the posterior probability of the vector $x$ with respect to the Gaussian $i$.
Any suggestions ? Thanks !
Best Answer
After a little research, I found what I was looking for, a paper called "Deriving cluster analytic distance functions from gaussian mixture models" which proposes an extension of the Mahalanobis distance in the context of multi-modal data (a GMM representation) using Fisher Kernel method and other techniques.