The first part of the hint follows from the Riemann-Lebesgue lemma (which I will state). We'll sketch the proof here as it is essentially the answer you are looking for. You should be able to coerce this into seeing why the second statement is true (the difference is that instead of having the cancellation of $ \sin(nx) $, $ \sin^2(nx) $ fills up half roughly half the space of a rectangle).
The Riemann-Lebesgue lemma: If $ f \in L^1(\mathbb R) $, then
$$
\int_{\mathbb R} f(x)\sin(nx)\,dx \to 0
$$
as $ n\to\infty $.
To prove the lemma, recall that step functions with compact support are dense in $ L^1(\mathbb R) $. Thus using dominated convergence it is enough to prove the result for such functions. However for a step function, the result follows from the straightforward calculation
$$
\int_a^b \sin(nx)\,dx = -\frac 1n(\cos(bn)-\cos(an))
$$
The analogous statement for the second part is
$$
\int_{\mathbb R} f(x)\sin^2(nx)\,dx \to \frac 12 \int_{\mathbb R} f(x)\,dx
$$
Applying these results to $ \chi_A $, the characteristic function of $ A $, we obtain the statements of the hint. Note that replacing $ n $ by $ n_k $ does not change the argument at all, we simply need $ n_k $ to go off to infinity.
A few ideas to take away from this:
- Instead of attempting to prove results directly, establish them first for "nice" objects and then hope "nice" objects are dense
- Instead of integrating $ f $ over $ A $, integrate $ f\,\chi_A $ over the entire domain
P.S. I'd leave this as a comment, but cannot due to low reputation: there is a typo in the hint as you've typed it.
For any measurable set $E \subset \mathbb{R},$ we have $\displaystyle \int_Ef \leqslant \int_{\mathbb{R}}f < \infty$ and $\displaystyle \int_{E}f_n \leqslant \int_{\mathbb{R}}f_n \to \int_{\mathbb{R}}f$.
Hence, for sufficiently large $n$, we have
$$\displaystyle \int_Ef_n < \infty.$$
Using Fatou's Lemma,
$$\int_E f = \int_E \lim f_n=\int_E \liminf f_n \leqslant \liminf \int_E f_n$$
and reverse Fatou's Lemma,
$$\liminf \int_E f_n \leqslant \limsup\int_E f_n \leqslant \int_E \limsup f_n = \int_E \lim f_n = \int_Ef.$$
Hence,
$$\tag{*}\int_Ef \leqslant \liminf \int_E f_n \leqslant \limsup\int_E f_n \leqslant\int_Ef,$$
and
$$\liminf \int_E f_n = \limsup\int_E f_n=\lim \int_E f_n = \int_Ef.$$
Update:
We can avoid the argument based on reverse Fatou as follows.
Note that since $f_n \to f$ and $\int_{\mathbb{R}} f_n \to \int_{\mathbb{R}} f$ we have
$$\begin{align}\limsup \int_E f_n &= -\liminf \left(-\int_E f_n \right) \\&= -\liminf \left(\int_{\mathbb{R}\setminus E} f_n-\int_{\mathbb{R}} f_n \right) \\ &= -\liminf \int _{\mathbb{R}\setminus E} f_n+\liminf \int_{\mathbb{R}} f_n \\ &\leqslant -\int_{\mathbb{R} \setminus E} \liminf f_n + \int_{\mathbb{R}} f \\ &= -\int_{\mathbb{R} \setminus E} f + \int_{\mathbb{R}} f \\ &= \int_E f\end{align} $$
The chain of inequalities (*) now follows.
Best Answer
Note that $\bigcap_n X \setminus K_n=\{x:|u(x)|=+\infty\}$ and since $u$ is integrable we have that $u$ is finite a.e so $\mu(\{x:f(x)=\infty\})=0$
Also $u_n(x):=|u(x)|1_{X\setminus K_n}(x) \to |u(x)|1_{\{x:f(x)=\infty\}}(x)$
So from DCT $$\int|u(x)|1_{X\setminus K_n}(x)d\mu(x) \to \int |u(x)|1_{\{x:f(x)=\infty\}}(x)d\mu(x)$$ $$=\int_{\{x:f(x)=\infty\}}|u(x)|d\mu(x)=0$$ since $|u| \geq 0$