Consider a Markov chain $\{X_n\}_{n\ge0}$ with state space
$S=\{1, 2, 3, 4, 5\}$, with the following transition matrix:
$\begin{bmatrix}\frac13&0&\frac13&\frac13 &0\\ \frac14&\frac14&0&\frac14&\frac14\\ \frac23&0&\frac13&0&0\\ 0&0&\frac13&\frac23&0\\ 0&0&0&0&1\end{bmatrix}$
Describe the transient and the recurrent states of the Markov chain. What
are the closed irreducible sets? Assume that $X_0 = 2$. Compute the distribution of $ T=\min\limits_n\{X_n=5\}\cup\{\infty\}$. That is, compute $F^{(n)}_{2,5}$ for all $n\ge0$
In general $F^{(n)}_{x,y}$ is the probability $P_x\left(X_n=y:X_i\neq y\ \text{for}\ i\in\{1,\dots,n-1\}\right)$ (i.e. first visit in state $y$ occurs at time $n$)
I found out that the states $\{1,3,4\}$ are closed and irreducible, state $2$ is transient, and since state $5$ is absorbing it is trivially recurrent.
$F^{(1)}_{2,5}=P(X_1=5|X_0=2)=\frac14$ (just read off the value from the matrix)
Since state $5$ is accessible only from state $2$
$F^{(n)}_{2,5}=\left(\prod\limits_{k=0}^{n-2}P(X_{k+1}=2|X_k=2)\right)P(X_n=5|X_{n-1}=2)$
When I exponentiate matrix it would give weird values, am I doing something wrong ?
Best Answer
Your calculations are right. Where do you need to exponentiate matrix? $$F^{(n)}_{2,5}=\left(\prod\limits_{k=0}^{n-2}\underbrace{P(X_{k+1}=2|X_k=2)}_{\frac14}\right)\cdot\underbrace{P(X_n=5|X_{n-1}=2)}_{\frac14}=\left(\frac14\right)^{n-1}\cdot \frac14=\frac{1}{4^n}.$$ Since $$ \mathbb P(T<\infty)=\sum_{n=1}^\infty \frac1{4^n} = \frac13, $$ then $T=\infty$ with the rest probability $\frac23$.