I have the following markov chain :
$S=\{0,1,2,3\}$
$p_{i,0} = q$ (if we are in one of the states $0,1,2,3$ we can return to $0$ with probability $q$)
$p_{i,i+1} = 1-q , i\in\{0,1,2\}$ (if we are in state $i$ we can move to state i+1 with probability $1-q$)
$p_{3,3} = 1-q$ (state $3$ can move to itself with probability $1-q$)
$0<q<1$
Since this chain is finite and irreducible, either all states are recurrent or transient, so I only need to prove that one state is recurrent. I tried calculating $P^n$ . I found that the eigenvalues of $P$ are $1,0,0,0$, but I don't know how to proceed (or if there is a better way to prove that the state is recurrent?)
Any help will be welcomed.
Best Answer
Define $\tau_{00} = \inf \{ n>0 : X_n = 0 | X_0=0 \}$ (with the usual convention $\inf \emptyset = +\infty$). Then either you stay at $0$ the first step, in which case $\tau_{00}=1$, or you leave $0$ in the first step, in which case $\tau_{00}=1+T$, where $T$ is geometric with parameter $q$. So $\mathbb{E}[\tau_{00}] \leq 1+\frac{1}{q}$. Now you use the strong Markov property to prove that this implies that $0$ is recurrent.