I am not sure how to put the question. I am not even sure if this question makes sense at all.
I know that the similarity of two discrete (or continuous) distributions can be quantified by Kullback–Leibler distance. However, I wonder if it makes sense to quantify the Kullback–Leibler distance between two random variables which one is discrete and the other one is continuous?
Is there any probabilistic measure for quantifying the similarity of continuos distribution with a discrete one.
Best Answer
There exist a lot of statistical distances for two distributions. http://en.wikipedia.org/wiki/Statistical_distance What motivated you to use KL convergence? Although it is often intuited as a metric or distance, the KL divergence is not a true metric — for example, it is not symmetric: the KL divergence from P to Q is generally not the same as that from Q to P.
But if you want to extend the definition of the KL divergence to the case when $X$ is continuous and $Y$ is discrete I think I know how to do it.
Any discrete PMF (probability measure function) can be represented as a continuous PDF using a Dirac delta function. http://en.wikipedia.org/wiki/Dirac_delta_function