markov chainsprobabilitystochastic-processes
A coin is tossed continuously until 2 heads or 2 tails appear respectively. Let the result of first toss is tail. The game is over when we get 2 heads respectively.
Determine the transition probability matrix.
Check my answer is correct or not.
Let $X_n$ denote the number of tail that appear.
Let the state $S=\{0,1,2\}$.
So, the transition probability matrix is
$$P=
\begin{bmatrix}
1&0&0\\
0&\dfrac{1}{2}&\dfrac{1}{2}\\
0&0&1
\end{bmatrix}.
$$
Determining the stable (long term) state probabilities
The solutions of the following system of equations are the stationary state probabilities:
$$\begin{matrix}[\pi_A\ \pi_B \ \pi_C]\\ \\ \\ \end{matrix} \begin{bmatrix} 0.2 & 0.4 & 0.4 \\ 0.4 & 0.3 & 0.3 \\ 0.6 & 0.2 & 0.2 \\ \end{bmatrix} \begin{matrix}=[\pi_A\ \pi_B \ \pi_C]\\ \\ \\\end{matrix} $$ and $$ \pi_A+\pi_B+\pi_C=1 .$$
Note
Here $\pi_A\not=1$. $\pi_A$ is a hypothetical quantity an intuitive definition of which (together with that of $\pi_B$,$\pi_C$) is as follows: "If we started the system choosing a state randomly with probabilities $\pi_A,\pi_B, \pi_C$ then the probability that the next state would be $A$, $B$, or $C$ would be $\pi_A,\ \pi_B, \pi_C$, respectively." Such stationary probabilities do not necessarily exists. You can read about the details here.
Continued
Having performed the vector-matrix multiplication we arrive at a redundant system of equations. After omitting the redundant equation and after some rearrangement we have
$$\begin{matrix} -0.8\pi_A & +& 0.4\pi_B & +& 0.6\pi_C &=0 \\ \ \ 0.4\pi_A & -& 0.7\pi_B& +& 0.2\pi_C&=0 \\ \pi_A&+&\pi_B&+& \pi_C&=1& \end{matrix}$$
The solution, with the help of Alpha, is $$\pi_A=\frac{5}{13}, \ \pi_B=\frac{4}{13},\ \pi_C=\frac{4}{13} .$$
Question a.
We can toss heads in each states, so $$P(\text{heads})=P(\text{heads}|\text{in state} A)\pi_A+P(\text{heads}|\text{in state}B)\pi_B+P(\text{heads}|\text{in state}C)\pi_C=\frac{2}{10}\frac{5}{13}+\frac{4}{10}\frac{4}{13}+\frac{6}{10}\frac{4}{13}=\frac{5}{13}=\pi_A.$$ That is, the result of the tossing is a head in $\frac{5}{13}100$% of the time on the long run.
Remark
It seemed to be accidental that
$$P(\text{heads})=\pi_A.$$
I wondered, if it was? So I took $p_A, p_B, \text{ and }p_C$ as the probabilities of the heads in the states $A, B, \text{ and } C$, respectively.
In order to get the stationary probabilities we have to solve the following system of equations:
$$\begin{matrix} p_A\pi_A & +& p_B\pi_B & +& p_C\pi_C &=\pi_A \\ \frac{1-p_A}{2}\pi_A & +& \frac{1-p_B}{2}\pi_B& +& \frac{1-p_C}{2}\pi_C&=\pi_B \\ \pi_A&+&\pi_B&+& \pi_C&=1& \end{matrix}.$$ The solutions are
$$ \begin{matrix} \pi_A=\frac{p_B+p_C}{-2p_A+p_B+p_C+2}\\ \pi_B=\frac{1-p_A}{-2p_A+p_B+p_C+2}\\ \pi_C=\frac{1-p_A}{-2p_A+p_B+p_C+2} \end{matrix}. $$ At this point, note that $p_A<1$. If $p_A$ was $1$ then the Markov chain would never get out of state $A$. Then, of course, $$1=P(\text{heads})=\pi_A.$$
If $p_A<1$ then the solutions above are valid and we may compute the probability of the heads in general. Watch this:
$$P(\text{heads})=\frac{(p_B+p_C)p_A+(1-p_A)p_B+(1-p_A)p_C}{-2p_A+p_B+p_C+2}=\pi_A \ !$$
Perhaps this is what the OOOOOP wanted to see.
Question b.
We have to go back to the initial state which is $A$. Now, the following vector-matrix product will give the probabilities that we end up in states $A$,$B$, or $C$ at the $\text{n}^{th}$ minute:
$$\begin{matrix}[1\ 0 \ 0]\\ \\ \\\end{matrix} \begin{bmatrix} 0.2 & 0.4 & 0.4 \\ 0.4 & 0.3 & 0.3 \\ 0.6 & 0.2 & 0.2 \\ \end{bmatrix}^n\begin{matrix}=[\pi_A^n\ \pi_B^n \ \pi_C^n]\\ \\ \\\end{matrix} $$
Finally, we have to repeat the steps performed during solution a. But this time we work with the results of the vector-matrix product above with $n=3$ and with $n=10$.
As I wrote above, there are eight states, which can be represented in this graph:
The numbers in each vertex describe the number of heads, then tails and whether the heads or the tails should be counted for the next transition. (Be sure the transitions away from any vertex sum to 1.0.)
It is a simple matter to label each edge by the probabilities of transition (head counting to tail counting or vice versa).
Best Answer
I think in this case you need to have 4 states: state 2 tails, state tail, state head and state 2 heads which I will call states 1,2,3 and 4 respectively.
$$P= \begin{bmatrix} 1&0&0&0\\ \dfrac{1}{2}&0&\dfrac{1}{2}&0\\ 0&\dfrac{1}{2}&0&\dfrac{1}{2}\\ 0&0&0&1 \end{bmatrix}. $$
If I understand you correctly you keep playing until you get either two heads or two tails. Then, states 1 and 4 are absorbing states.
Then when you are in state 2 (got a tail) then you can jump after the coin flip to either state 1 (two tail flips) or state 3 (one head).
If you start with a tail then your initial state probability vector $\pi$ would be
$$\pi= \begin{bmatrix} 0\\ 1\\ 0\\ 0 \end{bmatrix}. $$