We wish to evaluate the closed form for this integral,
$$I=\int_{0}^{\infty}\sin(x)\sin(ax)\ln(x)\cdot \frac{\mathrm dx}{x^2}$$
Let $a\ge1$
Using $2\sin(a)\sin(b)=\cos(a-b)-\cos(a-b)$
$$2I=\int_{0}^{\infty}\cos[x(a-1)]\ln(x)\cdot \frac{\mathrm dx}{x^2}-\int_{0}^{\infty}\cos[x(a+1)]\ln(x)\cdot \frac{\mathrm dx}{x^2}=I_1+I_2$$
$$I_1=-\frac{[1+\ln(x)]\cos[x(a-1)]}{x}+(a-1)\int [1+\ln(x)]\sin[x(a-1)]\frac{\mathrm dx}{x}$$
I don't think this is the right way to tackle this problem. How about another way?
Best Answer
Let $f(s)$ be defined by
$$ f(s) = \int_{0}^{\infty} \frac{1-\cos x}{x^s} \, \mathrm{d}x $$
for $1 < s < 3$. By writing $\frac{1}{x^s} = \frac{1}{\Gamma(s)} \int_{0}^{\infty} t^{s-1}e^{-xt} \, \mathrm{d}t$, we get
\begin{align*} f(s) &= \int_{0}^{\infty} \frac{t^{s-1}}{\Gamma(s)} \int_{0}^{\infty} (1 - \cos x)e^{-xt} \, \mathrm{d}x \mathrm{d}t \\ &= \frac{1}{\Gamma(s)} \int_{0}^{\infty} \frac{t^{s-2}}{1+t^2} \,\mathrm{d}t \\ &= \frac{1}{2\Gamma(s)} \int_{0}^{\infty} \frac{u^{(s-3)/2}}{1+u} \,\mathrm{d}u \qquad (u = t^2) \\ &= \frac{1}{2\Gamma(s)} \cdot \frac{\Gamma\big(\frac{s-1}{2}\big)\Gamma\big(\frac{3-s}{2}\big)}{\Gamma(1)} \\ &= -\frac{\pi}{2\Gamma(s)\cos(\pi s/2)}. \end{align*}
Substituting $x \mapsto \alpha x$ for $\alpha > 0$, this gives
$$ \int_{0}^{\infty} \frac{1-\cos (\alpha x)}{x^s} \, \mathrm{d}x = -\frac{\pi \alpha^{s-1}}{2\Gamma(s)\cos(\pi s/2)} $$
Differentiating $f$ in two ways, one using Leibniz's integral rule and the other using the above formula, we get
$$ \int_{0}^{\infty} \frac{1-\cos(\alpha x)}{x^s}\log x \, \mathrm{d}x = -\frac{\pi\alpha^{s-1}}{2\Gamma(s)\cos(\pi s/2)}\left(\psi(s) + \frac{\pi}{2}\tan(\pi s/2) - \log \alpha\right) $$
and in particular,
$$ \int_{0}^{\infty} \frac{1-\cos(\alpha x)}{x^2}\log x \, \mathrm{d}x = \frac{\pi\alpha}{2}\left(1 - \gamma - \log \alpha\right). $$
Together with OP's computation, this immediately gives the closed form of $I$.