Solved – Treating the log-likelihood as a probability distribution and normalizing

likelihoodnormalization

I have modeled a distribution, $f$, over a r.v. $x \in \mathbb{R}^3$. At inference a set of measuring points, $X$, of the r.v. variables show up. I want to form a distribution over this sample set so that I can sample from it so that more probable points will show up more often.

My idea was to evaluate $f$ for each point in the set and then normalize it to form a discrete distribution over the points. Since $f \approx 0$ I want to use the log likelihood. However, the $\log$ is a nonlinear transform and normalizing it means calculating:

$\log \sum_i f(x_i)$,

which is bound to give numerical errors due to the fact that $f(x_i)\approx 0$. So the best I can do is a log sum exp calculation.

My question is: is there a better way of doing this? And is this a completely wrong approach in terms of modeling probability distributions?

Best Answer

I'm not sure I understand things correctly. Here are my thoughts:

  1. If you have formed a likelihood for your data based on an assumption that each observation $x_i$ was a draw from a probability distribution which you have specified, then to generate similar samples you can just draw more samples from that distribution. So for example if you think each $x_i \sim MVN_3(0,I)$ then you can just draw more samples from a multi-variate Normal distribution (using R or some other Statistical software).
  2. If, on the other hand, you have a set of samples where each $x_i \sim f$, and you don't know what $f$ is, then it might be a good idea to look into Bootstrap re-sampling (see here: http://www.burns-stat.com/documents/tutorials/the-statistical-bootstrap-and-other-resampling-methods-2/ for a tutorial and here: http://en.wikipedia.org/wiki/Bootstrap_(statistics) for the wiki). Essentially you are drawing samples with replacement from your data, with a suitable degree of smoothing if necessary, which sounds a little like what you are describing. You can use the "sample" function in R (for example) to do this quite easily.

Does that help?

Related Question