For discrete time finite Markov chain, the first passage time $T_j$ to visit state $j$, is determined from the recurrence equation:
$$
p^{(n)}_{ij} = \sum_{k=0}^n f_{ij}^{(k)} p^{(n-k)}_{jj} =
\sum_{k=0}^n \mathbb{P}(T_j = k ; X_0 = i) \mathbb{P}(X_{n-k} = j ; X_0 = j)
$$
where $p^{(n)}_{ij} = \mathbb{P}(X_n = j ; X_0 = i)$ and $f^{(k)}_{ij} = \mathbb{P}(T_j = k ; X_0 = i)$. This recurrence equation allows to find probability generating function for the first passage time distribution (exerices 1.5.3 of J.R. Norris's book on "Markov Chains", relevant chapter 1.5 is available from Norris's website).
It strikes me odd that no book I have seen discusses first passage time distribution for continuous time finite Markov chain.
Can anyone suggest a reference where this is discussed/worked-out, or hint me at how to do this for myself.
I suspect a similar integral equation would be set-up, and moment generating function for the first-passage-time distribution would satisfy some simple equation.
Thanks for reading.
Best Answer
First of all, I wonder if $T_j$ refers to the first return time, or to the first hitting time. Since if you're talking about the latter case then $\mathsf P\{T_j = n-k|X_0 = j\} = 1_{\{k=n\}}$.
About continuous time case some insights are the following. I know results only for first hitting time. Let us introduce CTMC (continuous time Markov Chain) through its finite state space $\mathscr X$, transition matrix $R:\mathscr X\times \mathscr X\to[0,1]$ and exit-rate function $r:\mathscr X\to[0,\infty)$. For any initial point $x\in \mathscr X$ and target point $y\in \mathscr X$, the time-bounded reachability probability (or a cumulative distribution function for the first passage time) which we denote as $$ F(x,y,t) = \mathsf P_x\{\tau_y \leq t\} $$ is a least solution for $$ F(x,y,t) = 1_{\{y=x\}}+1_{\{y\neq x\}}\sum\limits_{z\in \mathscr X}R(x,z)\int\limits_0^t\mathrm e^{-r(x)s}F(z,y,t-s)\mathrm ds. $$
Assuming that $F$ has a density $f$ and recalling that $F(y,y,t) = 1$ we obtain: $$ f(x,y,t) = R(x,y)\mathrm e^{-r(x)t}+\sum\limits_{z\neq y}R(x,z)\int\limits_0^t\mathrm e^{-r(x)s}f(z,y,t-s)\mathrm ds $$ for any $x\neq y$.
With regards to the references - the only person whom I've found to work on that topic is he. The equation above I saw in his paper, but I do not remember which one. The problem may be also that his paper are focused on verification of general properties, usually much more complex that just reachability and it may be not so easy to go through the notation. However, I haven't seen something I can suggest you in the classical literature on Probability Theory. If you find it, please tell me.
Regards,