We have that $\sin(x)-x\cos(x)$ behaves like $x^3$ in a right neighbourhood of the origin, hence integrability over there is ensured by $\color{red}{\alpha < 4}$. By Dirichlet's test,
$$ \int_{1}^{+\infty}\frac{\sin x}{x^\beta}\,dx,\qquad \int_{1}^{+\infty}\frac{\cos x}{x^\beta}\,dx $$
are convergent as soon as $\color{red}{\beta>0}$, hence the original integral is convergent as soon as $\color{red}{\alpha\in(1,4)}$.
If you are allowed to use Laplace transforms,
you may also compute the value of the integral, since
$$ \mathcal{L}\left(\sin x-x\cos x\right) = \frac{2}{(1+s^2)^2}, \tag{1}$$
$$ \mathcal{L}^{-1}\!\!\left(x^{-\alpha}\right) = \frac{s^{\alpha-1}}{\Gamma(\alpha)} \tag{2}$$
and the equivalent integral
$$ I(\alpha)=\frac{1}{\Gamma(\alpha)}\int_{0}^{+\infty}\frac{s^{\alpha-1}}{(1+s^2)^2}\,ds \tag{3}$$
is convergent iff ${\alpha >0}$ (that is needed to grant integrability in a right neighbourhood of the origin) and ${\alpha < 4}$ (that is needed to grant integrability in a left neighbourhood of $+\infty$). In such a case, through the substitution $\frac{1}{1+s^2}=u$, Euler's beta function and the $\Gamma$ reflection formula we have:
$$ I(\alpha)= \color{red}{\frac{\pi\,(2-\alpha)}{2\,\Gamma(\alpha)\,\sin\left(\frac{\pi\alpha}{2}\right)}}.\tag{4}$$
$\newcommand{\bbx}[1]{\,\bbox[15px,border:1px groove navy]{\displaystyle{#1}}\,}
\newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
\newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\ds}[1]{\displaystyle{#1}}
\newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
\newcommand{\ic}{\mathrm{i}}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\mrm}[1]{\mathrm{#1}}
\newcommand{\on}[1]{\operatorname{#1}}
\newcommand{\pars}[1]{\left(\,{#1}\,\right)}
\newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
\newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
\newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
\newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$
\begin{align}
&\bbox[5px,#ffd]{\left.\int_{0}^{\infty}
{x^{1/2} \over 1-x^{2}}
{\sin\pars{ax}\sin\pars{a\bracks{1 - x}} \over 2}\,\dd x\,\right\vert_{\ a\ >\ 0}}
\\[5mm] = &\
\int_{0}^{\infty}
{x^{1/2} \over 1 - x^{2}}
{\cos\pars{2ax - a} - \cos\pars{a} \over 2}\,\dd x
\\[5mm] = &\
{1 \over 2}\,\Re\int_{0}^{\infty}
{x^{1/2} \over 1 - x^{2}}
\bracks{\expo{\ic\pars{2ax - a}} - \expo{\ic a}}\,\dd x
\\[5mm] = &\
{1 \over 2}\,\Re\bracks{\expo{-\ic a}\int_{0}^{\infty}
{x^{1/2} \over 1 - x^{2}}
\pars{\expo{2\ic ax} - \expo{2\ic a}}\,\dd x}
\\[5mm] \stackrel{x\ \mapsto\ x^{2}}{=}\,\,\, &\
\Re\bracks{\expo{-\ic a}\int_{0}^{\infty}
{x^{2} \over 1 - x^{4}}
\pars{\expo{2\ic ax^{2}} - \expo{2\ic a}}
\,\dd x}
\end{align}
Now,I'll "close" the integration along a pizza-slice $\ds{\mathcal{P}_{s}}$ in the first quadrant. Namely,
$\ds{\mathcal{P}_{s} =
\pars{0,R}\cup R\expo{\ic\pars{0,\pi/4}}\cup
\pars{R,0}\expo{\ic\pi/4}}$ with $\ds{R \to \infty}$. The integration along the arc $\ds{R\expo{\ic\pars{0,\pi/4}}}$ vanishes out as $\ds{R \to \infty}$.
Then,
\begin{align}
&\bbox[5px,#ffd]{\left.\int_{0}^{\infty}
{x^{1/2} \over 1-x^{2}}
{\sin\pars{ax}\sin\pars{a\bracks{1 - x}} \over 2}\,\dd x\,\right\vert_{\ a\ >\ 0}}
\\[5mm] = &\
-\Re\bracks{\expo{-\ic a}\int_{\infty}^{0}
{\ic r^{2} \over 1 + r^{4}}
\pars{\expo{-2ar^{2}} - \expo{2\ic a}}
\expo{\ic\pi/4}\dd r}
\\[5mm] = &\
\sin\pars{a - {\pi \over 4}}\
\underbrace{\int_{0}^{\infty}{r^{2}\expo{-2ar^{2}} \over r^{4} + 1}\dd r}
_{\ds{\equiv \mathcal{I}}}
\\[2mm] + &\
\sin\pars{a + {\pi \over 4}}\
\underbrace{\int_{0}^{\infty}{r^{2} \over r^{4} + 1}\dd r}
_{\ds{{\root{2} \over 4}\pi}}\label{1}\tag{1}
\end{align}
Lets evaluate $\ds{\mathcal{I}}$:
\begin{align}
\mathcal{I} & \equiv
\int_{0}^{\infty}{r^{2}\expo{-2ar^{2}} \over r^{4} + 1}\dd r =
\Re\int_{0}^{\infty}{\expo{-2ar^{2}} \over
r^{2} + \ic}\dd r
\\[5mm] & \stackrel{2ar^{2}\ \mapsto\ r^{2}}{=}
\,\,\,
\Re\int_{0}^{\infty}{\expo{-2ar^{2}} \over
r^{2} + \ic}\dd r =
\root{2a}\Re\int_{0}^{\infty}{\expo{-r^{2}} \over
r^{2} + 2a\ic}\dd r
\\[5mm] & =
\root{2a}
\Re\int_{0}^{\infty}\expo{-r^{2}}
\\[2mm] & \pars{%
{1 \over r - \root{2a}\expo{3\pi\ic/4}} -
{1 \over r + \root{2a}\expo{3\pi\ic/4}}}{1 \over 2\root{2a}\expo{3\pi\ic/4}}\dd r
\\[2mm] & =
{1 \over 2}\Re\braces{\expo{-3\pi\ic/4}
\bracks{\on{G}\pars{-\root{2a}\expo{3\pi\ic/4}} -
\on{G}\pars{\root{2a}\expo{3\pi\ic/4}}}}
\end{align}
where $\ds{\on{G}}$ is the
Goodwin-Staton Integral.
Finally ( see \ref{1} ),
\begin{align}
&\bbox[5px,#ffd]{\left.\int_{0}^{\infty}
{x^{1/2} \over 1-x^{2}}
{\sin\pars{ax}\sin\pars{a\bracks{1 - x}} \over 2}\,\dd x\,\right\vert_{\ a\ >\ 0}}
\\[5mm] = &\
{1 \over 2}\sin\pars{a - {\pi \over 4}}\ \times
\\[2mm] &\
\Re\braces{\expo{-3\pi\ic/4}
\bracks{\on{G}\pars{-\root{2a}\expo{3\pi\ic/4}} -
\on{G}\pars{\root{2a}\expo{3\pi\ic/4}}}}
\\[2mm] + &\
{\root{2} \over 4}\pi
\sin\pars{a + {\pi \over 4}}
\end{align}
Best Answer
What you have done is absolutely correct. In the last step, you can write $$-\int_{x}^\infty{\frac{\cos t}{t}\mathrm{d}t} = \operatorname{Ci}(x)$$ Where $\operatorname{Ci}(x)$ is the cosine integral of $x$. Otherwise, you can use numerical approximations to get the answer without finding the antiderivative.