Solved – Can Someone Explain How Factor Multiplication Works with Factor Graphs

Question

I'm taking the Probablistic Graphical Model course here: https://class.coursera.org/pgm-003/

This class uses the concept of Factors extensively with regards to graphical models: http://en.wikipedia.org/wiki/Factor_graph

I'm learning about making Clique Trees, http://en.wikipedia.org/wiki/Tree_decomposition, and part of that includes multiplying factors together to create new clique potentials, which will be used for inference later when you want to find marginal probabilities.

So, if you have:

F(A,B) and F(C | D) , you can make a clique with scope C(A, B, C, D) by using factor multiplication with F(A,B) and F(C | D)

And that's where I get lost. Can you just multiply conditional and joint probabilities like that? And then end up with a new joint probability? I would really appreciate if someone could explain how that's done.

Answer 1

To factor a joint probability, keep these in mind. Start with the definition of conditional probability:

$p(a|b) = \frac{p(a,b)}{p(b)}$

To make this a little more useful, multiply both sides by $p(b)$:

$p(a,b) = p(a|b)p(b)$

When the two are independent, $p(a|b) = p(a)$, and $p(a,b) = p(a)p(b)$. And finally, the definition of conditional independence: $p(a,b|c) = p(a|c)p(b|c)$

With all that in mind, one only needs to glean assumed dependencies from the graph. This tutorial gives a very nice example. From the second-to-last paragraph of p. 11:

As an example, assume a probability distribution $p(x_1, x_2, y)$ to factorize as $p(\boldsymbol x) = p(x_1)p(x_2)p(y|x_1, x_2)$. It has the factors $\Psi_1(x_1) = p(x_1), \Psi_2(x_2) = p(x_2)$, and $\Psi_3(y) = p(y|x_1, x_2)$. Here, $x_1$ and $x_2$ are conditionally independent given y. Figure 2 shows an independency graph and a factor graph representing this distribution.

There's also an illustration on the following page that might help intuit how probabilities relate to graphs. Final tip, I usually find it helpful to write the conditional probabilities first, before those of the given variables. E.g., these following are (obviously) equivalent, but I find the latter easier to read: $p(x_1)p(x_2)p(y|x_1, x_2) = p(y|x_1, x_2)p(x_1)p(x_2)$.