Solved – Calculating Emission Probability values for Hidden Markov Model (HMM)

Question

I'm new to HMM and still learning. I'm currently using HMM to tag part-of-speech. To implement the viterbi algorithm I need transition probabilities ($ a_{i,j} \newcommand{\Count}{\text{Count}}$) and emission probabilities ($ b_i(o) $).

I'm generating values for these probabilities using supervised learning method where I give a sentence and its tagging. I calculate emission probabilities as:

$$
b_i(o) = \frac{\Count(i \to o)}{\Count(i)}
$$

where $\Count(i)$ is the number of times tag $i$ occurs in the training set and $\Count(i \to o)$ is the number of times where the observed word $o$ maps to the tag $i$.

But when using this trained $b_i(o)$ for tagging, there might be observed variables in the given sentence that never appeared when finding the value for $b_i$. In such a case how do you estimate a value for $b_i$ for that instance?

Answer 1

For these kind of questions, it is possible to use Laplace Smoothing. In general Laplace Smoothing can be written as: $$ \text{If } y \in \begin{Bmatrix} 1,2,...,k\end{Bmatrix} \text{then,}\\ P(y=j)=\frac{\sum_{i=1}^{m} L\begin{Bmatrix} y^{i}=j \end{Bmatrix} + 1}{m+k} $$

Here $L$ is the likelihood.

So in this case the emission probability values ( $b_i(o)$ ) can be re-written as: $$ b_i(o) = \frac{\Count(i \to o) + 1}{\Count(i) + n} $$

where $n$ is the number of tags available after the system is trained.