I tried using integration by parts twice, the same way we do for $\int \sin {(\sqrt{x})}$
but in the second integral, I'm not getting an expression that is equal to $\int x\sin {(\sqrt{x})}$.
I let $\sqrt x = t$ thus,
$$\int t^2 \cdot \sin({t})\cdot 2t dt = 2\int t^3\sin(t)dt = 2[(-\cos(t)\cdot t^3 + \int 3t^2\cos(t))] = 2[-\cos(t)\cdot t^3+(\sin(t)\cdot 3t^3 – \int 6t \cdot \sin(t))]]$$
which I can't find useful.
Best Answer
Yes, indeed, continue as you did in the comments, treating $\int 6t\sin t \,dt\;$ as a separate integral, use integration by parts, and add (or subtract, if appropriate) that result to your earlier work, and you will end with an expression with no integrals remaining!:
$$\int t^2 \cdot \sin({t})\cdot 2t dt = $$
$$= 2[-\cos(\sqrt x) \cdot x(\sqrt x) + \sin(\sqrt x)\cdot 3x -(\cos(\sqrt x)\cdot6\sqrt x+\sin(\sqrt x)\cdot \sqrt x + \cos (\sqrt x))] + C$$
after substituting $\sqrt x$ for $t$, though I'd suggest finding a way to simplify (combining like terms, etc.)