Here's my transition matrix with state $\{1,2,3,4\}$:
\begin{bmatrix}
1/3 & 1/3 & 0 &1/3\\
1/4 & 1/4 & 1/4 &1/4\\
0 & 0& 1/2 &1/2\\
0 &0 &1/2 &1/2
\end{bmatrix}
I try to answer the following questions: If the Markov process starts from 2, what is the expected time that it visits state 3 or 4 in the first time? Moreover, what is the expected time that it visits the state 3 seconds time?
If I let $u_i$ be the expected time that it visits state 3 or 4 in the first time starting from i. Then from
$$
u_1=1+(1/3)u_2+(1/3)u_3, u_2=1+(1/4)u_1+(1/4)u_3+(1/4)u_4, u_3=u_4=0
$$
we can solve $u_2$ for the first part.
But what is the expected time that it visits the state 3 seconds time? General, how about visiting the state 3 for fourth time?
Best Answer
HINT: If a walk is started at 2, for it to visit 3 a second time, it must:
Since expectation is linear, the time you want is the sum of the expectations of those two events, which you can find by methods like what you outlined in your question.
EDIT to clarify based on your comments: I think you're still making this much too complicated.
First, a correction to your question: you outlined a solution to find $u_2$ but I think your system of equations had some mistakes. It should have been:
\begin{align*} u_1 &= 1 + \color{red}{\frac 1 3 u_1} + \frac 1 3 u_2 + \frac 1 3 u_4 \\ u_2 &= 1 + \frac 1 4 u_1 + \color{red}{\frac 1 4 u_2} + \frac 1 4 u_3 + \color{red}{\frac 1 4 u_4} \\ u_3 & = 0 \\ u_4 & = 0 \end{align*}
The intuition for why these relationships is valid is that from each state, you first take a single step, then weight the expected time to go from your first-step destination to 3 by the probability of each move. You can omit the $u_3$ and $u_4$ terms if you want since they're $0$, but the missing $u_1$ and $u_2$ terms will be important.
To find the expected time for the second visit somewhere, you'll need to solve some different systems of equations. If we take your example of starting the walk at 2 and computing its second visit to 3, we break that up into two occurrences; first, the walk must go from 2 to 3 a first time (taking however many steps it needs), then it must go from 3 to 3. You referenced a concern about "many cases" in your comments, but there are no cases; this is the only way to visit 3 twice.
To find the expected number of steps to go from 2 to 3, we set up a system very much like the one you outlined above:
\begin{align*} u_1 &= 1 + \frac 1 3 u_1 + \frac 1 3 u_2 + \frac 1 3 u_4 \\ u_2 &= 1 + \frac 1 4 u_1 + \frac 1 4 u_2 + \frac 1 4 u_3 + \frac 1 4 u_4 \\ u_3 &= 0 \\ u_4 &= 1 + \frac 1 2 u_3 + \frac 1 2 u_4 \end{align*} and note that these values of $u_1$ are different than the ones above. Solving for $u_2$ gives the first needed component.
For returning from 3 to 3, we need another system:
\begin{align*} u'_1 &= 1 + \frac 1 3 u'_1 + \frac 1 3 u'_2 + \frac 1 3 u'_4 \\ u'_2 &= 1 + \frac 1 4 u'_1 + \frac 1 4 u'_2 + \frac 1 4 u'_3 + \frac 1 4 u_4' \\ u'_3 &= \color{blue}{\frac 1 2 \cdot 1 + \frac 1 2 u'_4} \\ u'_4 &= 1 + \frac 1 2 u'_3 + \frac 1 2 u'_4 \end{align*}
We had to adjust the $u_3$ line here so that we guarantee that the walk takes at least one step. The intuition for the blue line is that a walk started at 3 will either immediately return to 3 (in which case the excursion required just 1 move), or it will go to state 4. Note that since we only care about $u'_3$, the $u'_1$ and $u'_2$ lines are irrelevant.
Solving the first system for $u_2$ gives the expected number of moves for a walk started at 2 to visit 3. Solving the second system for $u_3'$ gives the expected number of moves for a walk started at 3 to return to 3. Adding these together gives the expected number of steps for a walk started at 2 to make its second visit to 3. Again, there are no cases to consider, because everything is baked into the existing expected values we computed.