Given the transitional probabilities below (states: 0,1,2,3), I need to prove that state 3 is positive recurrent by calculating expected # of transitions to return to this state
$$P =
\pmatrix{0.65& 0.35 &0 &0\\
0.5& 0.1& 0.4 &0 \\
0.1 &0 &0 &0.9\\
0.6 &0 &0 &0.4\\}$$
now I see the all states communicate and the matrix is finite, then the MC is irreducible
and there is only one class. it also follows that the states are positive recurrent.
however, I need to make the claim about the positive recurrence of state 3 through calculating expected # of transitions to return to this state if starting from state 3.
How would I calculate this? I know there are some formulas for special Markov Chains, but I can't seem to understand the correct approach here. I cannot calculate it as E[..] by conditioning since it becomes infinite recurrence..
Best Answer
You can solve the following system of recursive relations. Denote by $h(k)$=the number of transitions to return to state $3$ when starting from state $k$. We want to calculate $h(3)$ and show that $h(3)<\infty$. Assume you start in state 3 as required and you will make a transition. You will either land in state $0$ with probability $0.6$ and start over or you can land in state $3$ with probability $0.4$ and in that case you are done. So the expected number of transitions $h(3)$ is equal to $$h(3)=1+0.6h(0)+0.4\cdot0$$ where $1$ stands for the transition that you already made. Now, we need to calculate $h(0)$. For that end, assume now that you start in state $0$ and you will make a transition. You will either land in state $0$ with probability $0.65$ or you can land in state $1$ with probability $0.35$ and start over. So the expected number of transitions $h(0)$ is equal to $$h(0)=1+0.65h(0)+0.35h(1)$$ Similarly we find the following system $$\begin{cases} h(3)=1+0.6h(0)+0.4\cdot0 \\ h(0)=1+0.65h(0)+0.35h(1) \\h(1)=1+0.5h(0)+0.1h(1)+0.4h(2) \\ h(2)=1+0.1h(0)+0.9h(3)\end{cases}$$ Solving the aboves yields $h(0)=30.1, h(1)=27.2 ,h(2)=21.2 ,h(3)=19$ (Doublecheck please for mistakes in the equations and the calculations, methodology is correct). So $$h(3)=19<\infty$$ and therefore state $3$ is positive recurrent (as is of course any other state due to the properties of communication that you have already mentioned).