I have a finite state and time-homogeneous continuous-time Markov chain (CTMC) which is not irreducible. Will steady state probabilities exist for this CTMC? How to prove this?
Solved – Steady state probabilities for a continuous-time Markov chain
markov-processprobabilitystochastic-processes
Related Solutions
I write this post as author of markovchain package. On August 2016 I pulled a fix to the package that should close the issue. Basically, the above transition matrix (TM) was composed by more than one closed class. Numerical issues could arise when solving the eigenvalue problem. So, we have decided to find the steady state distribution by closed class and to merge togheter. HTH
According to Wikipedia,
A state $i$ is accessible from a state $j$ (written $j\to i$) if a system started in state $j$ has a non-zero probability of transitioning into state $i$ at some point.
A state $i$ is essential if for all $j$ such that $i \to j$ it is also true that $j \to i$.
A state $i$ is said to be transient if, given that we start in state $i$, there is a non-zero probability that we will never return to $i$.
A Markov chain with an essential transient state can be constructed from three states $i,j,k$ for which $i\to j$, $j\to i$, $j\to k$, and $k$ never returns to $i$. The transition $j\to k$ guarantees $i$ is transient.
The transition matrix is
$$\pmatrix{0 & 1 & 0\\ 1-\rho & 0 & \rho \\ 0 & 0 & 1}$$
for some number $\rho$ with $0\lt \rho \lt 1$. It is the chance of never returning to $i$ when starting at $i$.
Best Answer
The short answer is "No."
First, it would be helpful to know if your underlying discrete-time Markov chain is aperiodic, unless you are using the phrase "steady state probabilities" loosely to mean "long-run proportion of the time the CTMC is in the various states" or something else other than "stationary distribution." Aperiodicity in combination with irreducibility is sufficient to guarantee a unique stationary distribution in the case of finite state-spaces, which you are assuming.
Second, the lack of irreducibility means that you will have either some transient states and/or more than 1 closed communicating class. Each class will have its own steady state probabilities (given aperiodicity) and there may be stationary distributions that span multiple communicating classes. Which class you wind up in depends upon what happened during the transient part of the chain's operation, and perhaps upon the initial state.
Consider a two-state discrete-time MC with transition matrix $P(1,1) = P(2,2)=1, P(1,2)=P(2,1) = 0$. Clearly it is aperiodic but not irreducible. Any steady-state distribution $\pi$ satisfies $\pi = \pi P$, is nonnegative, and sums to one. Obviously this is satisfied for any $\pi$ that is nonnegative and sums to one. So every distribution on $\{1,2\}$ is a steady-state distribution for this example. (Hence, clearly, a unique one does not exist.)