Since $t \wedge \tau_a$ is a bounded stopping time, a direct application of the optional stopping theorem shows that $\mathbb{E}(B_{t \wedge \tau_a}) = \mathbb{E}(B_0)=0$, i.e. $\mathbb{E}(X_t)=0$ for all $t \geq 0$.
If you do not want to use the optional stopping theorem, then there are several possibilities to compute the expectation but as far as I can see the hint, which you were given, does not work. As
$$X_t = B_{t \wedge \tau_a} = a 1_{\{\tau_a \leq t\}} + B_t 1_{\{\tau_a>t\}}$$
we have
$$\mathbb{E}(X_t) = a \mathbb{P}(\tau_a \leq t) + \mathbb{E}(B_t 1_{\{\tau_a>t\}}). $$
Using that $\{\tau_a>t\} = \{M_t < a\}$ for $M_t := \sup_{s \leq t} B_s$, we get
$$\mathbb{E}(X_t) = a \mathbb{P}(\tau_a \leq t) + \mathbb{E}(B_t 1_{\{M_t < a\}}). \tag{1}$$
If we could replace the second term on the right-hand side by $\mathbb{E}(B_t 1_{\{B_t<a\}})$, then we could differentiate this identity with respect to $t$ to get the equation which you were given as a hint. However, I don't see why it should possible to replace the second term. Here are two alternative approaches:
Approach 1 via strong Markov property
As $\mathbb{E}(B_t)=0$ for all $t \geq 0$, we have
$$\mathbb{E}(B_t 1_{\{M_t<a\}}) = - \mathbb{E}(B_t 1_{\{M_t \geq a\}})$$
and so
$$\mathbb{E}(B_t 1_{\{M_t<a\}}) = - \mathbb{E}(B_t 1_{\{\tau_a \leq t\}}).$$
Using the tower property of conditional expectation and the strong Markov property of Brownian motion, we get
$$\begin{align*} \mathbb{E}(B_t 1_{\{M_t<a\}})&= - \mathbb{E} \bigg( \mathbb{E}(B_t 1_{\{\tau_a \leq t\}} \mid \mathcal{F}_{\tau}) \bigg) = - \mathbb{E} \bigg( 1_{\{\tau_a \leq t\}} \mathbb{E}^{B_{\tau_a}}(B_{t-s}) \big|_{s=\tau_a} \bigg). \tag{2} \end{align*}$$
Since $$\mathbb{E}^{B_{\tau_a}}(B_r) =\mathbb{E}^a(B_r) = \mathbb{E}(a+B_r) = a$$
for any $r \geq 0$, we conclude from $(2)$ that
$$\mathbb{E}(B_t 1_{\{M_t<a\}}) =- a \mathbb{P}(\tau_a \leq t). $$
Plugging this into $(1)$ we infer that $\mathbb{E}(X_t)=0$.
Approach 2 via joint density of $(B_t,M_t)$
It is known that the joint density of $(B_t,M_t)$ equals
$$q_t(x,y) = 1_{\{x<y\}} \frac{2(2y-x)}{\sqrt{2\pi t^3}} \exp \left(- \frac{(2y-x)^2}{2t} \right),$$
and therefore the second term on the right-hand side of $(1)$ equals
$$\begin{align*} \mathbb{E}(B_t 1_{\{M_t<a\}}) &=\frac{2}{\sqrt{2\pi t^3}} \int_{-\infty}^a \int_{-\infty}^y x (2y-x) \exp \left( - \frac{(2y-x)^2}{2t} \right) \, dx \, dy \\ &\stackrel{\text{Fubini}}{=} \frac{2}{\sqrt{2\pi t^3}} \int_{-\infty}^a x \int_{x}^a (2y-x) \exp \left( - \frac{(2y-x)^2}{2t} \right) \, dy \, dx. \end{align*}$$
The inner integral can be computed easily and we get
$$\begin{align*} \mathbb{E}(B_t 1_{\{M_t<a\}}) &= \frac{1}{\sqrt{2\pi t}} \int_{-\infty}^a x \left( \exp \left[ - \frac{x^2}{2t} \right] - \exp \left[ - \frac{(2a-x)^2}{2t} \right] \right) \, dx. \end{align*}$$
If we denote by $p_t$ the density of $B_t$ then we obtain
$$\begin{align*} \mathbb{E}(B_t 1_{\{M_t<a\}}) &= \int_{-\infty}^a x p_t(x) \, dx - 2a \int_{-\infty}^a p_t(2a-x) \, dx + \int_{-\infty}^a (2a-x) p_t(2a-x) \, dx. \end{align*}$$
Performing a simple change of variables ($z=2a-x$) we conclude that
$$\begin{align*} \mathbb{E}(B_t 1_{\{M_t<a\}}) &= \underbrace{\int_{-\infty}^{\infty} x p_t(x) \, dx}_{0} - 2a \int_{a}^{\infty} p_t(z) \, dz = - a \mathbb{P}(|B_t| > a). \end{align*}$$
By the reflection principle, we have $\mathbb{P}(|B_t| \geq a) = \mathbb{P}(M_t \geq a)$, and therefore our computation entails that the terms on the right-hand side of $(1)$ cancel each other, i.e. $\mathbb{E}(X_t)=0$.
Best Answer
It is true that $B_t$ is Gaussian with mean $0$ and variance $t$ for any deterministic time $t$. However, if $T$ is a stopping time, then $B_T$ is, in general, not Gaussian. In fact, there is some general theorem which says that for a large class of distributions $\mu$ there exists a stopping time $T$ such that $B_T$ has distribution $\mu$.
Consider, for instance, the stopping time in your question, i.e. $$T = \inf\{t; B_t = a \, \, \text{or} \, \, B_t = b\}$$ for $a<0<b$. Because of the continuity of the sample paths of Brownian motion, it follows that $B_T$ takes only the values $a$ and $b$. This already shows that $B_T$ cannot be Gaussian.
To calculate the moments of $B_T$ you can use the fact that $$\mathbb{P}(B_T = a) = \frac{b}{b-a} \quad \text{and} \quad \mathbb{P}(B_T = b) = \frac{a}{a-b}$$ which is a consequence of the optional stopping theorem applied to the martingale $(B_t)_{t \geq 0}$. This means that we can write down explicitly the distribution of $B_T$: $$B_T \sim \frac{b}{b-a} \delta_a + \frac{a}{a-b} \delta_b$$ where $\delta_x$ denotes the Dirac measure. This allows us, in particular, to calculate arbitrary moments of $B_T$: $$\begin{align*} \mathbb{E}(B_T^k) = \mathbb{E}(B_T^k 1_{\{B_T=a\}}) + \mathbb{E}(B_T^K 1_{\{B_T=b\}}) &= a^k \mathbb{P}(B_T = a) + b^k \mathbb{P}(B_T=k) \\ &= a^k \frac{b}{b-a} + b^k \frac{a}{a-b}. \end{align*}$$