Let $(X_n)_{n \ge0}$ be a time-homogeneous Markov chain with state space $X = \{0, 1\}$ and transition probabilities: $p(0|0) = 0.4, p(1|0) = 0.6, p(0|1) = 0.25, p(1|1) = 0.75$.
a) Draw the transition graph and write the transition matrix of the Markov chain.
b) Compute the stationary distribution $\nu$ of the Markov chain.
c) Let $t^0$ be the return time to $0$ after starting with $X_0 = 0$. Based on the transition graph,
compute $E(t^0 \mid X_0 = 0)$ as the sum of an infinite series and then find $\nu(0)$ according to
the Ergodic Theorem for Markov Chains. Compare it with what you got in part (b).
My Attempt:
a) The transition graph looks as follows:
b) The transition matrix is given by:
$$\begin{bmatrix}
0.4 &0.6 \\
0.25 &0.75
\end{bmatrix}$$
And I calculated the stationary distribution as $\begin{bmatrix}-0.3846 &-0.9231 \end{bmatrix}$.
c) I know how the expected value is defined as an infinite series, but I do not see how I shall compute it in this case. Could you explain that to me?
Best Answer
The way how I read the first part of c) is if someone is at O, what is the expected number of "steps" (each step takes the same amount of time) till he is back at O? It will take 1 step if he just loops around at probability $0.4$. It will take 2 steps if he goes first to I and then comes back (0.6*0.25). The probability for 3 steps is 0.6*0.75*0.25. So the expectation can thus be set up: $$0.4*1+0.6*0.25*2+0.6*0.75*0.25*3+0.6*0.75^2*0.25*4...$$ When you factor out $0.6*0.25$ we get: $0.6*0.25[0.75^0(2)+0.75(3)+0.75^2(4)+...]$ Here we have an infinite series. Let's look at $\frac{1}{1-x}=1+x+x^2+x^3+x^4....$ which is the Geometric Series. Taking a derivative, we get $\frac{1}{(1-x)^2}=1+2x+3x^2+4x^3+...$. Then divide by $x$ and move $1/x$ over to the left: $\frac{1}{(1-x)^2}-\frac{1}{x}=2+3x+4x^2+...$This is exactly what we need. Plug in $x=0.75$ gives $20$ and then times $0.6*0.25$ gives $3$ as the answer. Add it to the $0.4$ from the beginning, we have an expectation of $3.4$. That is, if a person moves from O, you can expect this person to be back at O in 3.4 steps (or "time-units"). Hope this helps for the rest of the problem