That is a really convoluted way of describing a steady state in my opinion.
The set of potential states a system can be in is called $S$. For example, $S$ could be "how many animals are there in your socks".
They seem to be describing a system that proceeds in discrete steps (discrete time markov). So a step could be going from one day to the next.
$\pi_k(x)$ is the probability of being in state $x$ on step $k$. So there might be a 10% probability that you have 4 kittens in your socks on Tuesday: $\pi_{\text{Tuesday}}(4) = .1$.
$P(x,y)$ is the probability that you proceed from state $x$ to state $y$. So if you have 3 kittens in your socks on one day, there is a 14% chance that you'll have 5 the next day. $P(3,5) = .14$.
Every step has a probability associated with being in a certain state. If your socks can hold at most $5$ kittens, then $\pi(0) + \pi(1) + \pi(2) + \pi(3) + \pi(4) + \pi(5) = 1$. That's just basic probability, the sum of all possibilities is 100%.
Knowing the transition probabilities $P$, and the probability of states of a certain step $\pi_k()$, then you can calculate the probability of states in the next step, $\pi_{k+1}()$. Specifically, $\pi_{k+1}(y) = \sum_{x \in S} \pi_{k}(x)P(x,y)$. "The probability of having 2 kittens in your sockets is the probability of having 0 kittens times the probability of transitioning from 0 to 2 plus the probability of having 1 times the probability of transitioning from 1 to 2 plus ...".
When $\forall x ~~\pi_{k+1}(x) = \pi_k(x)$, then that $\pi$ is called a steady state. In that state, the transitions don't change the probabilty of each state from step to step.
Example:
You can have at most 2 kittens in your socks.
- If you have zero kittens in your sockets, then the probability that you have 0 the next day is 10%, 1 is 10%, 2 is 80%
- If you have one kitten in your sockets, then the probability that you have 0 the next day is 10%, 1 is 20%, 2 is 70%
- If you have two kittens in your sockets, then the probability that you have 0 the next day is 30%, 1 is 30%, 2 is 40%
Then a steady state is :
$$\pi(0) = \frac{27}{128},\quad \pi(1) = \frac{15}{64},\quad \pi(2) = \frac{71}{128}$$
As you can check, using the given formula.
"Also, could someone please confirm that I'm correct in thinking that the notation $p_{ij}$
denotes the probability of the process moving from the state $i$ to the state $j$?"
$(*)$
Correct
"How would you do this? What is a stationary distribution,
with respect to this example?"
If the chain starts in state $3$ it stays there forever because according
to $(*)$ there is zero probability to move to another state.
Therefore
$\pi_{b} = ( 0, 0, 1, 0, 0)$
is an obvious stationary distribution.
If the chain starts in state $1$ or $2$ it stays there forever because according
to $(*)$ there is zero probability to move to another state.
If the chain starts in state $4$ or $5$ it stays there forever because according
to $(*)$ there is zero probability to move to another state.
Now you can treat these as two $2 \times 2$ matrices and use the result that a vector which fulfills:
$\mathbf{\hat{\pi}} \mathbf{P} = \mathbb{\hat{\pi}}$ $\:\:(**)$
is a stationary distribution.
So you solve these two sets of systems of equations to get the remaining stationary distributions. Here you also need to use that $\hat{\pi}$ is a probability vector;
that is, its components sum to one.
"I now understand that stationary distributions are to do with looking at what happens to the probabilities at each state within a Markov Chain when time becomes infinitely large"
You also have this theorem that can be good to know:
If the Markov chain is irreducible and aperiodic then
$\lim \limits_{n \to \infty} P^n = \hat{P}$
where $\hat{P}$ is a matrix whose rows are identical and
equal to the stationary distribution $\mathbb{\hat{\pi}}$
for the Markov chain defined by equation $(**)$.
Best Answer
Stationary Distributions:
Let $\mathbf{P}$ be the transition probability matrix of a homogeneous Markov chain $\{X_n, n \geq 0\}$. If there exists a probability vector $\mathbf{\pi}$ such that $$\mathbf{\pi} \mathbf{P} = \mathbf{\pi} \:\:\:\:\:\:\: (1)$$
then $\mathbf{\pi}$ is called a stationary distribution for the Markov chain. Equation $(1)$ indicates that a stationary distribution $\mathbf{\pi}$ is a (left) eigenvector of $\mathbf{P}$ with eigenvalue $1$. Note that any nonzero multiple of $\mathbf{\pi}$ is also an eigenvector of $\mathbf{P}$. But the stationary distribution $\mathbf{\pi}$ is fixed by being a probability vector; that is, its components sum to unity.
Limiting Distributions:
A Markov chain is called regular if there is a finite positive integer $m$ such that after $m$ time-steps, every state has a nonzero chance of being occupied, no matter what the initial state. Let $A > 0$ denote that every element $a_{ij}$ of $A$ satisfies the condition $a_{ij} > 0$. Then, for a regular Markov chain with transition probability matrix $\mathbf{P}$, there exists an $m > 0$ such that $\mathbf{P}^m > 0$. For a regular homogeneous Markov chain we have the following theorem: