Let $(X_t)$ be a continuous-time Markov chain and $\tau$ the first jump time. Compute $\mathbb E_x [a^{\tau} \phi (X_\tau)]$

expected valueintegrationmarkov chainsmarkov-processstochastic-processes

Let $(X_t)$ be a continuous-time Markov chain such that

  • The state space $V$ is finite and endowed with discrete topology.

  • The infintesimal generator is $L: V^2 \to \mathbb R$.

Let

  • $\alpha \in (0,1)$.

  • $\phi$ be a function from $V$ to $\mathbb R_+$.

  • $\tau$ is the first jump time, i.e. the first time that the chain makes a transition to a new state.

I would like to ask how to compute $$\alpha = \mathbb E_x [a^{\tau} \phi (X_\tau)]$$ where $\mathbb E_x := \mathbb E_x [ \cdot | X_0 = x ]$.

My attempt:

It's well-known that given $X_0$, $\tau$ is exponentially distributed with parameter $-L(X_0,X_0)$. Then

$$\alpha = \mathbb E_x [a^{\tau} \phi (X_\tau)] = -\int_0^\infty a^s L(x,x)\phi (X_s) e^{-sL(x,x)} \mathrm{d}s$$

I'm stuck because there is $s$ inside $\phi(X_s)$. Could you please elaborate on how to compute $\alpha$?

Thank you so much!

Best Answer

Thank you so much for @Saad's invaluable comment! I post it here to close this question:

The issue in the calculation is using the formula$$E(g(τ))=\int g(t)\,\mathrm dF_τ(t),$$ which requires that $g$ be a deterministic measurable function of $τ$, but $Χ_τ$ is not deterministically determined by $τ$.

I'm not so familiar with continuous-time Markov chains, but I think there should exist a result similar to that in How $h(z)=\color{blue}{\alpha}\sum_yp_{z y}h(y)$ follows from Markov property?.

Related Question