[Math] Modeling with Markov Chains and one-step analysis

probabilityprobability theorystochastic-processes

enter image description here

I have set up the following model:

Let $X_n$ be the number of heads in the $n$-th toss and $P(X_0=0)=1$. I can calculate the transition matrix $P$. Define
$$
T=\min\{n\geq 0\mid X_n=5\}.
$$
Then $P(X=1)=P(X_{T-1}=4)$. Noting that $X_n$ is a Markov chain and defining
$$
u_i=P(X_{T-1}=4\mid X_0=i)
$$
we have
$$
u_i=\sum_{j=0}^5P(X_{T-1}=4\mid X_1=j)\cdot P_{ij}.
$$
I ended up with $u_0=0$ which seems nonsense.

  • [Edited:]What is wrong with my model?

  • Would anybody come up with a better one?


[Added:] Here is possibly where I get the calculation wrong:

$$
P(X_{T-1}=4\mid X_1=i)=u_i
$$
and $u_5=0$.


[Last Update:] What is wrong is
$$
P(X_{T-1}=4\mid X_1=4)=u_4.
$$
One should get $P(X_{T-1}=4\mid X_1=4)=1$ instead.

Best Answer

It might simplify slightly to say we just require the chain to reach state $4$ at any point, starting at $0$, and we can forget about the variable $T$. So re-define $u_i$ as

$$u_i = P(X_n = 4 \text{ for some $n$}\mid X_0=i).$$

Our transition probability matrix:

$$ \begin{matrix} \qquad 0 \qquad & 1 \qquad & 2 \qquad & 3 \quad & 4 \qquad & 5 \\ \end{matrix} \\ P = \begin{bmatrix} 1/32 & 5/32 & 10/32 & 10/32 & 5/32 & 1/32 \\ 0 & 1/16 & 4/16 & 6/16 & 4/16 & 1/16 \\ 0 & 0 & 1/8 & 3/8 & 3/8 & 1/8 \\ 0 & 0 & 0 & 1/4 & 1/2 & 1/4 \\ 0 & 0 & 0 & 0 & 1/2 & 1/2 \\ 0 & 0 & 0 & 0 & 0 & 1 \\ \end{bmatrix} $$

The values for $u_i$ can be calculated in turn, using first-step analysis, until we get the required value, $u_0$:

\begin{align} u_3 &= \frac{1}{4}u_3 + \frac{1}{2} \\ \therefore\quad u_3 &= \frac{2}{3} \\ & \\ u_2 &= \frac{1}{8}u_2 + \frac{3}{8}u_3 + \frac{3}{8} \\ \therefore\quad u_2 &= \frac{5}{7} \\ & \\ u_1 &= \frac{1}{16}u_1 + \frac{1}{4}u_2 + \frac{3}{8}u_3 + \frac{1}{4} \\ \therefore\quad u_1 &= \frac{76}{105} \\ & \\ u_0 &= \frac{1}{32}u_0 + \frac{5}{32}u_1 + \frac{5}{16}u_2 + \frac{5}{16}u_3 + \frac{5}{32} \\ \therefore\quad u_0 &= \frac{157}{217}. \end{align}