[Math] Finding the steady state Markov chain

markov chains

I have drawn a certain Markov chain with a weird transition matrix. Here's the drawing:

https://dl.dropbox.com/u/106018191/markovpolicy.png

And here's the transition matrix:

https://dl.dropbox.com/u/106018191/mathstackex.png

My problem is that I don't quite know how to calculate the steady state probabilities of this chain, if it exists. I need a bit of help, because just getting the $\pi$'s, they don't sum to 1 for me, which is weird.

I've assembled this Markov chain using some data and conditions given to me and, right now, I would insist that it is the inevitable result of a very long process of solving. But now I hit a brick wall and can't continue. Please help?

Edit:

Here's my solution to the problem, using the ordinary ways of calculating steady states.

solution

As you can see, it's a little bit problematic. The probabilities don't add up to 1. I've turned it upside down several times. Is it really just something about my arithmetic, or is this simply a weird Markov chain?

Best Answer

There are two equations here: $\pi P=\pi$ and $\sum_i \pi_i=1$. So you will get eight equations to solve for seven variables. You need to omit one of the equations you get from $\pi P=\pi$ and solve the linear equations.

$$P=\left[\begin{array}{ccccccc}l&m&n&0&0&0&0\\o&p&0&0&0&0&0\\ o&p&0&0&0&0&0\\1&0&0&0&0&0&0\\1&0&0&0&0&0&0\\1&0&0&0&0&0&0\\1&0&0&0&0&0&0\end{array}\right]$$

These are the linear equations you need to solve:

$$l\pi_0+o(\pi_1+\pi_2)+\pi_3+\pi_4+\pi_5+\pi_6=\pi_0\\ m\pi_0+p(\pi_1+\pi_2)=\pi_1\\ n\pi_0=\pi_2\\ \pi_0+\pi_1+\pi_2+\pi_3+\pi_4+\pi_5+\pi_6=1$$

Related Question