I feel it goes to zero I separate the integral $\int_0^1 \frac{x^{n-2}\cos(n\pi x)}{1+x^n}$ the sequence goes to zero and it bounded by $0.5$ so by bounded convergence theorem the limit is zero. Now for the $\int_1^\infty \frac{x^{n-2}\cos(n\pi x)}{1+x^n} $ first $\frac{x^{n-2}}{1+x^n}\leq \frac{1}{x^2}$ which integrable. Another thought as $n$ goes large enough the $\int_1^\infty\cos(n\pi x)$ will be zero and the other part will be almost constant. Any help with full details.
Trying Partial derivative as @EuklidAlexandria suggested
$$\int_1^\infty \frac{x^{n-2} \cos(n\pi x) }{1+x^n} dx = 0 -\int_1^\infty \underbrace{\frac{x^{n-3}(n-2-2x^n)\sin(n\pi x)}{n\pi(1+x^n)^2}}_{f_n}$$
then the sequence
$$\left| f_n \right| \leq \left|\frac{x^{n-3}(n-2x^n)}{n\pi x^{2n}}\right|\leq \frac{n+2x^n}{n x^{n+3}}\leq \frac{1}{x^n} + \frac{1}{x^3}\leq \frac{2}{x^3}$$ since we are concern about limit we can consider $n\geq 3$
Now by Lebesgue Dominated convergence theorem we have $f_n \rightarrow 0$ p.w and $|f_n| \leq \frac{2}{x^3}$ which is integrable on $[1,\infty)$ Hence
$$\lim_{n\rightarrow \infty} \int_{1}^{\infty} \frac{x^{n-2}\cos(n\pi x)}{1+x^n} = \lim_{n\rightarrow \infty}- \int_1^\infty \frac{x^{n-3}(n-2-2x^n)\sin(n\pi x)}{n\pi(1+x^n)^2} =\int 0 = 0$$
Is that correct?
Best Answer
Your solution seems fine. For a bit of further simplification, we may do as follows. Write $I_n$ for the integral. Then
\begin{align*} &\left| I_n - \int_{1}^{\infty} \frac{\cos(n\pi x)}{x^{2}} \, \mathrm{d}x \right| \\ &\leq \int_{0}^{1} \left| \frac{x^{n-2}\cos(n\pi x)}{1+x^n}\right| \, \mathrm{d}x + \int_{1}^{\infty} \left| \frac{x^{n-2}}{1+x^n} - \frac{1}{x^2} \right|\left| \cos(n\pi x)\right| \, \mathrm{d}x \\ &\leq \int_{0}^{1} x^{n-2} \, \mathrm{d}x + \int_{1}^{\infty} \frac{1}{x^2(1 + x^n)} \, \mathrm{d}x. \end{align*}
It is easy to check that this bound converges to $0$ as $n\to\infty$. Now, to show that $I_n \to 0$, it suffices to check that
$$ \int_{1}^{\infty} \frac{\cos(n\pi x)}{x^{2}} \, \mathrm{d}x \xrightarrow[n\to\infty]{} 0, $$
which is an immediate consequence of the Riemann-Lebesgue lemma. (Or, of course, integration by parts works perfectly.)