Assume we inspect a system periodically for a certain time period. After each inspection, we classify the system states into exactly one of the states in $\left\{0, 1, \ldots, M\right\}$. State $0$ indicates the best condition of the machine, while state $L$ means the machine is inoperative. During every periods, the system state is likely to degrade by $1$ unit with probability $p$.
(a) Let $Y_n$ be the state of the system at time $n$. Determine the transition probability matrix $P$. Is $\left\{Y_n: n\geq 0\right\}$ ergodic?
(b) Given $0 < l^{*} < L$. Assume that if $Y_n > l^{*}$, the system is replaced by a new one (aka. back to state $0$ at time $n+1$ from time $n$). Find the new transition probability matrix $P^{*}$.
My attempt:
Based on the problem statement, I interpret that in each time period, the system can either degrade by $k$ unit lower with probability $kp$ ($k\in \left\{1, \ldots, M\right\}$), or stay at the same level with probability. Is this the intended interpretation?
Thus, for the two successive time periods $n$ and $n-1$, $P(Y_n = i | Y_{n-1} = j) = (i-j)p$ for $i > j$ (i.e., degrade by $(i-j)$ units). For $i = j$, $P(Y_n = i | Y_{n-1} = j) = P(Y_n = j | Y_{n-1} = j) = 1-\sum_{k=1}^{M} kp$ (no degrade). For $i<j$, $P(Y_n = i | Y_{n-1} = j) = 0$.
Thus, the one-step transition probability matrix is (bear in mind that the rows read as $0$ to $M$ from top to bottom, and the columns read as $0$ to $L$ from left to right):
$$
P=
\pmatrix{
1-p\sum_{k=1}^{M} k & p & 2p &\cdots & Mp\\
0 & (1 – p\sum_{k=1}^{M-1} k) &p &\cdots &(M-1)p\\
\vdots &\ddots &\ddots &\ddots &\vdots\\
0 &0 & \cdots & 1-p & p\\
0 &0 & 0 &\cdots & 1
}
$$
Looking at the transition probability matrix $P$, it is straightforward that $\left\{Y_n: n\geq 0\right\}$ is not ergodic, because we cannot find a $n$ such that $P_{Lj}^{n} > 0$ for $j = 1, 2, \ldots, L-1$.
(b) In this case, all the rows from $0$ to $l*$ of matrix $P*$ will be exactly the same as in
$P$, and all the rows after $l*$-th row will have the form $[1 \ 0 \ 0 \ldots 0$] corresponding to columns $0$ to $L$. Once again, from this matrix $P*$, we can show that $\left\{Y_n: n\geq 0\right\}$ is ergodic.
My question: Can anyone please let me know if my solutions above are correct?
Best Answer
I had started off by pointing out that the interpretation of the system is wrong. Indeed, in one step either there was no degradation, or degradation by one unit, therefore the transition matrix was far simpler than you wrote it (Look up Gambler's ruin as well). It is indeed not ergodic, since you can't return to $0$ starting from $1$, for example. This was subsequently noted by you.
Your transition probabilities are fine for the second one. However, the second system is also not ergodic , because for $l^* <k <L$, a system starting at $k$ goes to $0$, then cannot hit $k$ since in the middle it will hit $l^*$ and again go back to $0$.