Let's write your integral as $\DeclareMathOperator{re}{Re}$
$$
\int_{-\infty}^{\infty} f(t) e^{kg(t)}\,dt,
$$
where
$$
f(t) = \frac{1}{1+t^2} \qquad \text{and} \qquad g(t) = \frac{i}{5}t^5 + it.
$$
The lay of the land.
The critical points of the exponent function $g$ occur at $t = e^{i\pi (2k+1)/4}$, $k=0,1,2,3$.
Here's a plot showing these critical points in yellow with the paths of constant altitude (of $\re g$) passing through them shown in white. The real axis is shown in black. The background is colored according to the value of $\re g(t)$, with higher points colored lighter and lower points darker.
Note that the function $\re g$ has ten "hills" and "valleys" radiating away from the origin. Since
$$
g(t) \sim \frac{i}{5}t^5
$$
as $t \to \infty$ we deduce that these hills and valleys lie approximately on the rays
$$
t = s e^{i\pi(2k+1)/10}, \quad s > 0,\,k = 0,1,\ldots,9,
$$
with even $k$ corresponding to valleys and odd $k$ corresponding to hills.
In order for the contour to pass through either of the two saddle points in the lower half-plane we would need to deform at least one tail of the current contour (the real axis) over one of these hills. This doesn't seem feasible, so we'll instead focus on the two saddle points in the upper half-plane.
The new contour.
With a little work it's possible to show that we can deform the contour to the one shown in black in the following image. We'll call this new contour $\gamma$.
Here lines of constant altitude on the surface $\re g(t)$ are again shown in white. The point $t=i$ is shown in yellow.
The new contour $\gamma$ consists of two curves. The first originates at $t = e^{i\pi 9/10} \infty$ then passes through the saddle point at $t = e^{i\pi 3/4}$ at an angle of $\pi/8$ before terminating at $t = i \infty$. The second originates at $t = i \infty$, passes through the saddle point at $t = e^{i\pi/4}$ at an angle of $-\pi/8$, then terminates at $e^{i\pi/10} \infty$.
Note that to deform the contour from the real axis to $\gamma$ we must enclose the pole of $f$ located at $t=i$. Ultimately we have
$$
\begin{align}
\int_{-\infty}^{\infty} f(t) e^{kg(t)}\,dt &= \int_\gamma f(t) e^{kg(t)}\,dt + 2\pi i\operatorname{Res}\left(f(t) e^{kg(t)},t=i\right) \\
&= \int_\gamma f(t) e^{kg(t)}\,dt + \pi e^{-6k/5}. \tag{1}
\end{align}
$$
We will show in the next section that the term $\pi e^{-6k/5}$ is negligible compared to the integral $\int_\gamma$.
Estimating the new integral.
Now we will estimate the integral
$$
I(k) = \int_\gamma f(t) e^{kg(t)}\,dt.
$$
We've chosen the contour $\gamma$ in such a way that the largest values of $\re g(t)$ for $t \in \gamma$ occur at the saddle points $t = e^{i\pi 3/4}, e^{i\pi/4}$. Further,
$$
\re g\left(e^{i\pi 3/4}\right) = \re g\left(e^{i\pi/4}\right) = -\frac{4}{5\sqrt{2}}.
$$
Consequently we'll need to take both saddle points into account. For the first we have
$$
g\left(e^{i\pi 3/4} + se^{i\pi/8}\right) = \frac{4}{5} e^{-i\pi 3/4} - 2s^2 + O(s^3)
$$
and for the second we have
$$
g\left(e^{i\pi/4} + se^{-i\pi/8}\right) = \frac{4}{5} e^{i\pi 3/4} - 2s^2 + O(s^3)
$$
as $s \to 0$, so applying the Laplace method yields, to leading order,
$$
\begin{align}
I(k) &\approx e^{i\pi/8} f\left(e^{i\pi 3/4}\right) \int_{-\infty}^{\infty} \exp\left[k\left(\frac{4}{5} e^{-i\pi 3/4} - 2s^2\right)\right]\,ds \\
&\qquad + e^{-i\pi/8} f\left(e^{i\pi/4}\right) \int_{-\infty}^{\infty} \exp\left[k\left(\frac{4}{5} e^{i\pi 3/4} - 2s^2\right)\right]\,ds \\
&= \sqrt{\frac{\pi}{k}} \exp\left(-\frac{4}{5\sqrt2}k\right) \cos\left(\frac{4}{5\sqrt2}k - \frac{3\pi}{8}\right)
\end{align}
$$
as $k \to \infty$. Combining this with equation $(1)$ we conclude that, to leading order,
$$
\int_{-\infty}^{\infty} f(t) e^{kg(t)}\,dt \approx \sqrt{\frac{\pi}{k}} \exp\left(-\frac{4}{5\sqrt2}k\right) \cos\left(\frac{4}{5\sqrt2}k - \frac{3\pi}{8}\right)
$$
as $k \to \infty$, as desired.
Best Answer
In Bender & Orszag, at least in my edition (the first, 1978), this is problem 6.73 on p. 313. On p. 308, problem 6.19(a), you are asked how many terms one can get using integration by parts. (The answer is one - the second term blows up at the $t=0$ limit.) Your answer is correct for the leading term.
The idea here is to use steepest descents. I will outline here how to go about setting up the steepest descent contours and setting up the integrals used to derive the asymptotic expansions. Really, this case follows closely Example 2 of Sec. 6.6 in B & O.
Recall that the whole idea behind steepest descent is to find an approximation to the complex integral
$$\int_C dz \, h(z) e^{x \rho(z)} $$
as $x \to \infty$. We do this by deforming $C$ to a steepest descent contour $C'$ along which we may end up getting integrals to which we may apply Laplace's method, e.g., decaying exponentials. So, again, consider the integral of interest:
$$\int_0^{\pi/4} dt \, \cos{x t^2} \, \tan^2{t} = \operatorname{Re} \int_0^{\pi/4} dt \, e^{i x t^2} \, \tan^2{t}$$
Our contour $C$ is just a line segment over the real axis. What would be the steepest descent contour $C'$? It is that over which $\operatorname{Im}{\rho(z)}$ is constant. Letting $z=u+i v$, this would mean that we want a contour over which $i z^2 = -2 u v+i (u^2-v^2) $ is constant.
At $z=0$, $\operatorname{Im}{(i z^2)}=0$, so we can use a contour defined by $u=v$. ($u=-v$ defines a steepest ascent contour, which does not deliver an integral useful for Laplace's method.) Thus, the contour $C_1$ coming from $z=0$ is $z=e^{i \pi/4} t$.
We will discuss a contour $C_3$ coming from the other endpoint at $z=\pi/4$ in a minute. We deform the original integral into a closed contour $\gamma=C_1+C_2-C_3-C$ as follows:
The contour $C_2$ is the top piece and vanishes as we go farther along the contours $C_1$ and $C_3$. Thus, by Cauchy's theorem we have
$$\int_0^{\pi/4} dt \, e^{i x t^2} \tan^2{t} = \int_{C_1} dz \, e^{i x z^2} \tan^2{z} - \int_{C_3} dz \, e^{i x z^2} \tan^2{z} $$
Now,
$$\int_{C_1} dz \, e^{i x z^2} \tan^2{z} = e^{i \pi/4} \int_0^{\infty} dt \, e^{-x t^2} \tan^2{\left ( e^{i \pi/4} t \right )} $$
Now, at $z=\pi/4$, $\operatorname{Im}{(i z^2)}=\pi^2/16$. Thus, along $C_3$, $u^2-v^2=\pi^2/16$, or $u=\sqrt{v^2+\pi^2/16}$ and
$$\rho(z) = i z^2 = i \frac{\pi^2}{16} - 2 v \sqrt{v^2+\frac{\pi^2}{16}} $$
As in the book, let $s= 2 v \sqrt{v^2+\frac{\pi^2}{16}}$ and so on, and we find that
$$\int_{C_3} dz \, e^{i x z^2} \tan^2{z} = i e^{i \pi^2 x/16} \frac{2}{\pi} \int_0^{\infty} ds \, \left (1+i \frac{16 s}{\pi^2} \right )^{-1/2} \tan^2{\left (\frac{\pi^2}{16}+i s \right )^{1/2}} \, e^{-x s}$$
I hope you can see where this is going. We expand separately along each contour for large $x$. At $C_1$, i.e. $t=0$, it should be clear that
$$\begin{align} e^{i \pi/4} \int_0^{\infty} dt \, e^{-x t^2} \tan^2{\left ( e^{i \pi/4} t \right )} &= i e^{i \pi/4} \int_0^{\infty} dt \, e^{-x t^2} [t^2+O(t^3)]\\ &= -i e^{i \pi/4} \frac14 \sqrt{\pi} x^{-3/2} + O(x^{-2}) \end{align}$$
At $C_3$, i.e., $t=\pi/4$, we have
$$\begin{align} i e^{i \pi^2 x/16} \frac{2}{\pi} \int_0^{\infty} ds \, \left (1+i \frac{16 s}{\pi^2} \right )^{-1/2} \tan^2{\left (\frac{\pi^2}{16}+i s \right )^{1/2}} \, e^{-x s} &= i e^{i \pi^2 x/16} \frac{2}{\pi} \int_0^{\infty} ds \, \left [1+O(s)\right ] e^{-x s} \\ &= i e^{i \pi^2 x/16} \frac{2}{\pi x} + O(x^{-2}) \end{align}$$
Finally, subtracting the two contributions and taking the real part, we get the first two terms of the expansion:
The nice part is that we can keep adding on terms if we want.
Here is a graph verifying the above asymptotic expansion:
The blue plot is the log-log error between a numerical value of the integral and the first term in the expansion. The red plot is the log-log-error with both terms. Note that the first plot has a supremum that has slope -1.5, and the red has one that has slope -2.0. This indicates that the expansion above is correct.