I have a Markov chain with state space $S = \left\{ 1,2,\dots \right\}$ with transition matrix as follows
$\begin{bmatrix}
0 & 1 & 0 & 0 & 0 & 0 & \dots \\
0 & 0 & 1 & 0 & 0 & 0 & \dots \\
0 & 0 & \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\
0 & 0 & \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\
0 & 0 & \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\
\vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \ddots
\end{bmatrix}
$
I am trying to find $\lim_{n\to\infty} P^n(4,7)$
and I believe I should be using this theorem:
$\textbf{Theorem 7}$ Let $X_n, n > 0$, be an irreducible positive recurrent
Markov chain having stationary distribution $\pi$. If the chain is aperiodic,
$(55) \lim_{n\to\infty} P^n(x,y) \space x,y \in S$
If the chain is periodic with period d, then for each pair $x, y$ of states in
$S$ there is an integer $r, 0 \leq r < d$, such that $P^n(x, y) = 0$ unless
$n = md + r$ for some nonnegative integer $m$, and
$(56) \lim_{m\to\infty} P^{md+r}(x,y) = d\pi(y)$
The problem is that it does not satisfy the conditions of being irreducible and positive recurrent so I cannot use the theorem. However I see that if I consider only the block not containing states $1,2$ ie
$\begin{bmatrix}
\frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\
\frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\
\frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\
\vdots & \vdots & \vdots & \vdots & \ddots
\end{bmatrix}
$
Then this is irreducible, positive recurrent and would be aperiodic and can use the theorem. Would it be valid to apply the theorem to this block matrix so that I can find
$\lim_{n\to\infty} P^n(4,7)$?
Best Answer
Yes it is appropriate to consider the the state transition matrix
$$\begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix} $$
since the original chain gets in this chain in two steps. So we can forget about the first two states, 1 and 2, $P_1$ and $P_2$ will be $0$ on the long run because the chain will never get (back) to these states.
In the case of the transition matrix above, it is easy to calculate the stationary probabilities:
So, we have the stationary probabilities in a row vector $\pi=[P_3\ P_4 \ P_5\cdots]$ and we have the equation for the stationary probabilities
$$[P_3\ P_4 \ P_5\cdots]\begin{bmatrix} \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix}=[P_3\ P_4 \ P_5\cdots].$$
In the case of the row vector and the firs column we get
$$\frac23(P_3+P_4+\cdots)=\frac23=P_3.$$ Then in the case of the row vector and the second column we have
$$P_3\frac13=\frac29=P_4$$
and so on.
So we have for the row vector of the stationary probabilities
$$\left[0 \ 0\ \frac23\ \ \frac29\ \ \frac2{27}\ \cdots\right].$$
Here $P_1$ and $P_2$ were included.
If I understood the question well then
$$\lim_{n\to\infty} p^n(4,7)=\frac2{3^5}=\pi(7)$$
because
$$\lim_{n\to\infty} \begin{bmatrix} 0&1&0&0&\cdots\\ 0&0&1&0&\cdots\\ \frac{2}{3} & \frac{1}{3} & 0 & 0 & \dots \\ \frac{2}{3} & 0 & \frac{1}{3} & 0 & \dots \\ \frac{2}{3} & 0 & 0 & \frac{1}{3} & \dots \\ \vdots & \vdots & \vdots & \vdots & \ddots \end{bmatrix}^n =\begin{bmatrix} 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\frac2{3^5}\cdots\\ 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\frac2{3^5}\cdots\\ 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\frac2{3^5}\cdots\\ 0&0&\frac23&\frac2{3^2}&\frac2{3^3}&\frac2{3^4}&\boxed{\color{red}{\frac2{3^5}}}\cdots\\ \vdots & \vdots & \vdots & \vdots & \vdots& \vdots & \ddots \end{bmatrix}.$$