Consider the Markov Chain with state space $S=\{v,w,x,y,z\}$, transition matrix below:
$$\left[\begin{array}{cccccccccc}
0 & 0.4 & 0.6 & 0 & 0\\
0 & 0.5 & 0.5 & 0 & 0\\
0 & 0 & 0 & 0.1 & 0.9\\
0 & 0 & 0& 0.2 & 0.8\\
0.7 & 0 & 0.3 & 0 & 0
\end{array}\right]$$
(where the matrix rows/columns correspond to the states in alphabetical order), and $X_0=v$.
1) What is $P(X_1=x, X_2=z, X_3=v)$?
For this question, do I multiply all the states starting from state x -> state z -> state v?
which I get: $$0.9 \times 0.7 = 0.63$$ Am I on the right track?
2) Find the probability distribution of $X_4$.
Any hints for this question please?
3) Find $P(\{X_2=w\} \cup \{X_3=x\})$ and $P(X_2=x|X_4=z)$
Im completely getting lost. I'm assuming that the first one is finding the union of $X_2$ and $X_3$ and the second question is finding the probability of $X_2$ given $X_4$, I am not sure what these "X" values are. Anyone could you please help me or give me a hint on these questions?
Best Answer
First, let's agree on notation: by $X_n$ we mean the state at "time" $n$; and we are told that the initial state ($n=0$) is $X_0 = v$. The transition matrix, denotes $M_{i,j} = P(X_{n+1}=j | X_{n}=i)$ (the row corresponds to the 'before' state, the column to the 'after' state).
For the first question, you want to compute a particular transition path. But remember that you start from $X_0$ (it can help to draw a graph of the transitions), so you actually are computing:
$$P(X_1=x,X_2=z,X_3=v | X_0=v)=\\ =P(X_1=x | X_0=v) P(X_2=z|X_1=x) P(X_3=v|X_2=z) =\\ = 0.6 \times 0.9 \times 0.7$$
(The first equation is true because it's a Markov chain).
For the second, you need to compute the probabilities of arriving of each one of the five states at time $n=4$. You could do that by summing all the paths that start from $X_0=v$, but that would be painful. (In general, perhaps not so much in this case because there are few transitions with positive probability). A more elegant way is to recall that the 4-step transition probabilities is given by $M^4$. Once you compute that, you just take the first row.