In the step of learning Markov chains, I came across few questions from assignments on UTDallas website. Finding the transition probabilities seems a bit hard for me in this one particular question only. The question is
(Sec 7.3, page 355, #1) Consider a system with two components. We observe the
state of the system every hour. A given component operating at time $n$ has probability $p$ of failing
before the next observation at time $n + 1$. A component that was in a failed condition at time $n$
has a probability $r$ of being repaired by time $n + 1$, independent of how long the component has
been in a failed state. The component failures and repairs are mutually independent events. Let
$X_n$ be the number of components in operation at time $n$. The process $\{X_n, n = 0, 1, . . .\}$ is a
discrete time homogeneous Markov chain with state space $I = \{0, 1, 2\}$.
a) Determine its transition probability matrix, and draw the state diagram.
b) Obtain the steady state probability vector, if it exists
Although the answers are given, but I cannot understand that on what basis the transition probabilities are calculated. Can someone help me in this…
I had the following guesses my ownself (which mostly proved to be wrong)
\begin{bmatrix}
1-r-r^2 & r & r^2\\
??& ?? & r(1-p) \\
p^2 & ?? & ??
\end{bmatrix}
The actual solution that the website pose is following…
\begin{bmatrix}
(1-r)^2 & 2r(1-r) & r^2\\ p(1-r)& pr + (1-p)(1-r) & r(1-p) \\
p^2 & 2p(1-p) & (1-p)^2
\end{bmatrix}
Best Answer
Note that, there are two components and each of them individually could be either in working or failure state. Then the states of the Markov chain be designated as $ff$ both components failed, $wf$ one component working and one component failed, and $ww$ both components working. Given that, the components function independently of each other.
If the chain is in state $ff$ at time $n$, at the next time point it could be in the state
$ff$ with probability $(1-r)(1-r)=(1-r)^2$, as the probability of not being repaired by next time point is $(1-r)$;
$wf$ with probability $2r(1-r)$, as it may be the first component
that was repaired or the the second component that was repaired, so
that only one of the two components will be working;
$ww$ with probability $r^2$, if both the components got repaired.
If the chain is in the state $wf$ at time $n$, then at the next time point it could be in the state
$ff$ with probability $p(1-r)$, as the working component failed and the component requiring repair was not yet repaired;
$wf$ with probability $pr+(1-p)(1-r)$, as the working component was
not failed and the component requiring repair is not repaired, so
that only one component is working, which has a probability
$(1-p)(1-r)$; or the component working at the previous time has
failed and the component requiring repair has got repaired, which has a probability $pr$; mutually exclusiveness of the events results in the required probability;
$ww$ with probability $r(1-p)$, as the working component does not
require repair and the failed component got repaired.
If the chain is in the state $ww$ at time $n$, then at the next time point it could be in the state
$wf$ with probability $2p(1-p)$, which is the sum of probabilities
of two mutually exclusive events, viz., the first component failed
and the second working or the first component working and the
second component failed;
working.
Hence, the transition matrix:
\begin{equation*} P=\begin{array}{c|ccc} &ff & wf & ww\\ \hline ff & (1-r)^2& 2r(1-r) & r^2\\ wf & p(1-r) & pr+(1-p)(1-r) & r(1-p) \\ ww & p^2 & 2p(1-p) & (1-p)^2 \end{array} \end{equation*}