What is the difference between Cross Correlation and Mutual Information. What kind of problems can be solved using these measures and when is it appropriate to use one over the other.
Thanks for the comments. To clarify, the question is prompted by an interest in image analysis rather than time series analysis although any enlightenment in that area would also be appreciated
Best Answer
Cross correlation assumes a linear relationship between 2 sets of data. Whereas mutual information only assumes that one value of one dataset says something about the value of the other dataset.
So mutual information makes much weaker assumptions.
A traditional problem solved with mutual information is aligning (registration) of two types of medical images, for example an ultrasound and a x-ray image. (typically, the types of images are called modalities, so the problem is named multi-modal image registration).
For both X-ray and ultrasound, a specific material, say bone, leads to a certain 'brightness' in the image. Whereas some materials lead to a bright x-ray and ultrasound image, for other materials (e.g. fat) it might be the opposite, one is bright, the other is dark. Therefore, it is not the case that bright parts of the X-ray image are also bright parts of the ultrasound.
Therefore, mutual information is still a useful criterion for aligning the images, but cross correlation is not.