I play the following game: "I roll 2 dices, I win if the sum of the 2 dices is 5, I lose if the sum is 7, otherwise I roll them again" What is my probability of winning?
It can be easily shown the probability is 0.4.
However, I thought, such game is an example of a Markov chain and I also tried to solve it as such. Hence I defined 3 states: Win, Loss, Continue, the following transition matrix $T$:
$$
\begin{matrix}
1 & 0 & 4/36 \\
0 & 1 & 6/36 \\
0 & 0 & 26/36 \\
\end{matrix}
$$
and the following intial vector $X_{0}$:
$$
\begin{matrix}
0 \\
0 \\
1 \\
\end{matrix}
$$
I see that when n increase, $T^n.X_{0}$ converges as I expected to
$$
\begin{matrix}
0.4 \\
0.6 \\
0 \\
\end{matrix}
$$
However, when I tried to get the long-term stable states by looking at the eigenvalues, eigenvectors I did not find what I was looking for, i.e. I was expecting eigen value of 1 and eigen vector (0.4, 0.6, 0)
Can someone tell me whether or not this is a Markov Chain example. If it is, why I cannot find the long term solution by looking at the eigen values and vectors ?
Thank you in advance for your help
Best Answer
For there to be a unique stationary distribution, the Markov chain must be both irreducible and aperiodic. It fails the first requirement because of the two absorbing states: it’s not possible to get to any other state from them, let alone to every other state.
In terms of eigenvectors of the transition matrix, you should have found that the eigenspace of $1$ is two-dimensional, so even if you consider only eigenvectors with entries in $[0,1]$ that sum to $1$, there’s an infinite number of them.
You can, however, analyze it using standard techniques for absorbing Markov chains. For this process, the fundamental matrix $N$ is just the scalar $1/(1-26/36)=18/5$, and the absorption probabilities are $\frac{18}5\left(\frac19,\frac16\right)=\left(\frac25,\frac35\right)$, which agrees with your direct computation.