That is a really convoluted way of describing a steady state in my opinion.
The set of potential states a system can be in is called $S$. For example, $S$ could be "how many animals are there in your socks".
They seem to be describing a system that proceeds in discrete steps (discrete time markov). So a step could be going from one day to the next.
$\pi_k(x)$ is the probability of being in state $x$ on step $k$. So there might be a 10% probability that you have 4 kittens in your socks on Tuesday: $\pi_{\text{Tuesday}}(4) = .1$.
$P(x,y)$ is the probability that you proceed from state $x$ to state $y$. So if you have 3 kittens in your socks on one day, there is a 14% chance that you'll have 5 the next day. $P(3,5) = .14$.
Every step has a probability associated with being in a certain state. If your socks can hold at most $5$ kittens, then $\pi(0) + \pi(1) + \pi(2) + \pi(3) + \pi(4) + \pi(5) = 1$. That's just basic probability, the sum of all possibilities is 100%.
Knowing the transition probabilities $P$, and the probability of states of a certain step $\pi_k()$, then you can calculate the probability of states in the next step, $\pi_{k+1}()$. Specifically, $\pi_{k+1}(y) = \sum_{x \in S} \pi_{k}(x)P(x,y)$. "The probability of having 2 kittens in your sockets is the probability of having 0 kittens times the probability of transitioning from 0 to 2 plus the probability of having 1 times the probability of transitioning from 1 to 2 plus ...".
When $\forall x ~~\pi_{k+1}(x) = \pi_k(x)$, then that $\pi$ is called a steady state. In that state, the transitions don't change the probabilty of each state from step to step.
Example:
You can have at most 2 kittens in your socks.
- If you have zero kittens in your sockets, then the probability that you have 0 the next day is 10%, 1 is 10%, 2 is 80%
- If you have one kitten in your sockets, then the probability that you have 0 the next day is 10%, 1 is 20%, 2 is 70%
- If you have two kittens in your sockets, then the probability that you have 0 the next day is 30%, 1 is 30%, 2 is 40%
Then a steady state is :
$$\pi(0) = \frac{27}{128},\quad \pi(1) = \frac{15}{64},\quad \pi(2) = \frac{71}{128}$$
As you can check, using the given formula.
"Also, could someone please confirm that I'm correct in thinking that the notation $p_{ij}$
denotes the probability of the process moving from the state $i$ to the state $j$?"
$(*)$
Correct
"How would you do this? What is a stationary distribution,
with respect to this example?"
If the chain starts in state $3$ it stays there forever because according
to $(*)$ there is zero probability to move to another state.
Therefore
$\pi_{b} = ( 0, 0, 1, 0, 0)$
is an obvious stationary distribution.
If the chain starts in state $1$ or $2$ it stays there forever because according
to $(*)$ there is zero probability to move to another state.
If the chain starts in state $4$ or $5$ it stays there forever because according
to $(*)$ there is zero probability to move to another state.
Now you can treat these as two $2 \times 2$ matrices and use the result that a vector which fulfills:
$\mathbf{\hat{\pi}} \mathbf{P} = \mathbb{\hat{\pi}}$ $\:\:(**)$
is a stationary distribution.
So you solve these two sets of systems of equations to get the remaining stationary distributions. Here you also need to use that $\hat{\pi}$ is a probability vector;
that is, its components sum to one.
"I now understand that stationary distributions are to do with looking at what happens to the probabilities at each state within a Markov Chain when time becomes infinitely large"
You also have this theorem that can be good to know:
If the Markov chain is irreducible and aperiodic then
$\lim \limits_{n \to \infty} P^n = \hat{P}$
where $\hat{P}$ is a matrix whose rows are identical and
equal to the stationary distribution $\mathbb{\hat{\pi}}$
for the Markov chain defined by equation $(**)$.
Best Answer
Yes, the set of stationary distributions will always be convex, for the reason you give.
Some Markov chains (like simple random walk on the integers) have no stationary distributions. But finite MCs have stationary distributions, so your statement about "1 or infinitely many" is correct, in this case.