I am trying to use Bayes formula to compute the likelihood of a given state given a collection of independent but not sequenced observations – knowing the priors and knowing the probabilities of being in the state given each observation.
In other words for pre-defined states $S = \{S_1,\dots,S_m\}$ and possible observations $O=\{O_1,\dots,O_n\}$, the prior and conditional probabilities are already known:
$$ P(S_i) = \pi_i $$
$$P(S_i|O_j) = \rho_{ij}$$
(Note the second formula is not the "observation emission probability" but rather the state given observation probability.)
Suppose now I have collected a set of $k$ observations $\Theta=\{\theta_1,\dots, \theta_k\}$ and wish to compute for each state $i$:
$$P(S_i | \Theta) = P(S_i | \theta_1,\dots,\theta_k)$$
i.e. to work out the most likely state given the observations. I am having a brainfade as trying to use Bayes formula directly goes nowhere:
$$P(S_i | \theta_1,\dots,\theta_k) = \frac{P(\theta_1,\dots,\theta_k|S_i)P(S_i)}{\alpha}$$
I see the relevance of Hidden Markov Models (HMMs) in the mathematics but there is no natural sequence I'm dealing with here it is just a collection of unordered observations. What's the correct way to compute this?? (Pretend it's a HMM with a flat transition probability matrix??)
Suppose I accepted some independence assumptions and continued the derivation as follows:
$$ = \frac{P(\theta_1|S_i)\cdot \dots \cdot P(\theta_k|S_i)P(S_i)}{\alpha} $$
Then applied Bayes formula again to each term:
$$ = \frac{\frac{P(S_i|\theta_1)P(\theta_1)}{P(S_i)}\cdot \dots \cdot \frac{P(S_i|\theta_k)P(\theta_k)}{P(S_i)} P(S_i)}{\alpha} $$
Seems to be getting a bit silly and now I have to come up with $P(\theta_i)$ priors too (possible but surprised I need this) – what's going on and what's the correct way to compute this??
Best Answer
This is correct: \begin{align}\mathbb P(S=S_i|\theta_1,\ldots,\theta_k) \propto \frac{\mathbb P(S=S_i|\theta_1)p(\theta_1)}{\pi_i}\cdots \frac{\mathbb P(S=S_i|\theta_k)p(\theta_k)}{\pi_i} \pi_i\\ \propto \mathbb P(S=S_i|\theta_1)\cdots \mathbb P(S=S_i|\theta_k) \pi_i^{-k+1} \end{align} and the $p(\theta_j)$'s play no role there.