[Math] Find expected value of time to reach a state in Markov chain, by simulation

markov chainssimulation

Consider a time homogeneous Markov chain
$ (X_n)_{n=0} $
with state space $E$, initial distribution $p(0)$ and transition probability matrix
$P$ given by
$E = \{0, 1, 2\}, p(0) = [1\;\; 0\;\; 0]$ and

P= $\begin{bmatrix}1/2 & 1/3 & 1/6\\0 & 2/3 & 1/3 \\0 & 0 & 1 \end{bmatrix}$

respectively. Find by a computer simulations an as good as is possible approximation of the expected value $\Bbb E(T)$ of the time $T = \min\{ n ∈ N : X_n = 2 \}$ it takes the chain to reach the state 2. Someone who can help me in doing some kind of pseudocode for this question, or some matlab code?

Best Answer

In Mathematica:

myP = {{1/2, 1/3, 1/6}, {0, 2/3, 1/3}, {0, 0, 1}};

ListPlot[Transpose[NestList[#.myP &, {1, 0, 0}, 10]], 
 AxesLabel -> {"step", "Probability"},
 Joined -> True,
 PlotLegends -> {"P[0]", "P[1]", "P[2]"} ]

enter image description here

The successive differences in probabilities for being in state $2$ are:

difProbs = Differences@NestList[#.myP &, {1, 0, 0}, 100][[All, 3]];

The expected value is:

N@Range[100].difProbs

(* 4 *)

Related Question