Let $0<b<a$ and define $$J(a,b)=\int_0^1\frac{\arctan\sqrt{t^2+a}}{(t^2+b)\sqrt{t^2+a}}dt.\tag1$$
I am seeking a closed form for $J(a,b)$.
I was motivated to find a closed form for $(1)$ after seeing the approach taken in this answer and thinking that it could be generalized.
I have actually succeeded in finding a closed form for the special case $(a,b)=(a,\tfrac{a}{2})$:
$$J(a,\tfrac{a}{2})=\frac{\pi^2}{2a}-\frac{\pi}{a}\arctan\sqrt{a+1}-\frac1a\arctan^2\sqrt{\frac{2}{a}},$$
which comes from the more general result
$$J(a,b)+J(a,a-b)=\phi_2(a,b)-\phi_1(b)\phi_1(a-b)+\phi_2(a,a-b),\tag2$$
where $$\phi_1(x)=\int_0^1\frac{dt}{x+t^2}=\int_1^\infty\frac{dt}{1+xt^2}=\frac1{\sqrt x}\arctan\frac1{\sqrt x},$$
and $$\phi_2(a,b)=\frac{\pi}{2\sqrt{b(a-b)}}\arctan\sqrt{\frac{a-b}{b(a+1)}}.$$
I will supply a proof below.
Proof of $(2)$. Let $$f(z)=\int_0^1\frac{\arctan(z\sqrt{t^2+a})}{(t^2+b)\sqrt{t^2+a}}dt,$$
so that $J(a,b)=f(1)$. Then clearly $$J(a,b)=f(1)=\lim_{z\to\infty}f(z)-\int_1^\infty f'(z)dz.$$
Then since $\lim_{z\to\infty}\arctan(xz)=\pi/2$ for $x>0$, we have
$$J(a,b)=\frac\pi2\int_0^1\frac{dt}{(t^2+b)\sqrt{t^2+a}}-\int_1^\infty\int_0^1\frac{dt}{(t^2+b)(z^2t^2+az^2+1)}dz.$$
The first integral is relatively simple:
$$\begin{align}
\int_0^1\frac{dt}{(t^2+b)\sqrt{t^2+a}}&=\int_0^{1/\sqrt{a}}\frac{du}{(au^2+b)\sqrt{u^2+1}}\qquad[t\mapsto u\sqrt{a}]\\
&=\int_0^{\arctan 1/\sqrt{a}}\frac{\sec^2x\,dx}{(a\tan^2x+b)\sqrt{\tan^2x+1}}\qquad[u\mapsto \tan x]\\
&=\int_0^{\arctan 1/\sqrt{a}}\frac{\cos x\,dx}{b+(a-b)\sin^2x}\\
&=\int_0^{1/\sqrt{a+1}}\frac{dt}{b+(a-b)t^2}\qquad [\sin x\mapsto t]\\
&=\left.\frac{1}{\sqrt{b(a-b)}}\arctan\left(t\sqrt{\frac{a-b}{b}}\right)\right|_0^{1/\sqrt{a+1}}\\
&=\frac{1}{\sqrt{b(a-b)}}\arctan\sqrt{\frac{a-b}{b(a+1)}}=\frac2\pi\phi_2(a,b).\tag3
\end{align}$$
The next integral is also manageable:
$$\begin{align}
\int_1^\infty f'(z)dz&=\int_1^\infty\int_0^1\frac{dt}{(t^2+b)(z^2t^2+az^2+1)}dz\\
&=\int_1^\infty\int_0^1\frac{1}{1+(a-b)z^2}\left(\frac{1}{t^2+b}-\frac{1}{t^2+a+1/z^2}\right)dtdz\\
&=\int_1^\infty\frac{1}{1+(a-b)z^2}\left(\phi_1(b)-\phi_1(a+1/z^2)\right)dz\\
&=\phi_1(b)\phi_1(a-b)-\int_1^\infty \frac{\arctan(1/\sqrt{a+1/z^2})}{(1+(a-b)z^2)\sqrt{a+1/z^2}}dz\\
&=\phi_1(b)\phi_1(a-b)-\int_0^1 \frac{\arctan(1/\sqrt{t^2+a})}{(t^2+(a-b))\sqrt{t^2+a}}dt\qquad [z\mapsto 1/t]\\
&=\phi_1(b)\phi_1(a-b)-\int_0^1 \frac{\tfrac\pi2-\arctan\sqrt{t^2+a}}{(t^2+(a-b))\sqrt{t^2+a}}dt\\
&=\phi_1(b)\phi_1(a-b)-\frac\pi2\int_0^1 \frac{dt}{(t^2+(a-b))\sqrt{t^2+a}}+\int_0^1 \frac{\arctan\sqrt{t^2+a}}{(t^2+(a-b))\sqrt{t^2+a}}dt\\
&=\phi_1(b)\phi_1(a-b)-\phi_2(a,a-b)+J(a,a-b).\tag4
\end{align}$$
Then from $(3)$ and $(4)$,
$$J(a,b)=\phi_2(a,b)-\phi_1(b)\phi_1(a-b)+\phi_2(a,a-b)-J(a,a-b),$$
as desired.
Is there some way to find a closed form for $J$?
EDIT:
One may use the series
$$g(z)=\frac{\arctan\sqrt z}{\sqrt z}=\sum_{n\ge1}\frac{(-1)^n}{2n-1}\left(\frac{1}{z^n}-\frac{2}{\sqrt z}\right)\qquad z\ge1$$
to get
$$J(a,b)=-\phi_2(a,b)+\sum_{n\ge1}\frac{(-1)^n}{2n-1}\int_0^1\frac{dt}{(t^2+b)(t^2+a)^n},$$
where $\phi_2$ is defined above. The integral $\int_0^1\frac{dt}{(t^2+b)(t^2+a)^n}$ is really not nice though.
Best Answer
Define the function $\mathcal{I}:\mathbb{R}_{>0}^{2}\rightarrow\mathbb{R}$ via the definite integral
$$\mathcal{I}{\left(p,q\right)}:=\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\sqrt{t^{2}+p^{2}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}.\tag{1}$$
We seek a closed-form expression for $\mathcal{I}{\left(p,q\right)}$ for $\left(p,q\right)\in\mathbb{R}_{>0}^{2}$ such that $p>q$. It should be clear this problem is completely equivalent to the one posed by the OP.
We will make use of the following Euler substitution:
$$\sqrt{t^{2}+p^{2}}=t+x;~~~\small{p>0\land t\in\mathbb{R}\land x\in\mathbb{R}_{>0}}.\tag{2}$$
Solving for $t$, we have
$$t=\frac{p^{2}-x^{2}}{2x},$$
$$\implies\sqrt{t^{2}+p^{2}}=t+x=\frac{p^{2}+x^{2}}{2x},$$
$$\implies dt=dx\,\frac{(-1)\left(p^{2}+x^{2}\right)}{2x^{2}}.$$
Suppose $\left(p,q\right)\in\mathbb{R}_{>0}^{2}\land p>q$. We find
$$\begin{align} \mathcal{I}{\left(p,q\right)} &=\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\sqrt{t^{2}+p^{2}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\int_{0}^{1}\mathrm{d}t\,\frac{\left[\frac{\pi}{2}-\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}\right]}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{1}\mathrm{d}t\,\frac{1}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{1}{\sqrt{1+p^{2}}}}\mathrm{d}u\,\frac{1}{\left(1-u^{2}\right)}\cdot\frac{1}{\left(\frac{pu}{\sqrt{1-u^{2}}}\right)^{2}+q^{2}};~~~\small{\left[t=\frac{pu}{\sqrt{1-u^{2}}}\right]}\\ &~~~~~-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{1}{\sqrt{1+p^{2}}}}\mathrm{d}u\,\frac{1}{p^{2}u^{2}+q^{2}\left(1-u^{2}\right)}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{1}{\sqrt{1+p^{2}}}}\mathrm{d}u\,\frac{1}{q^{2}+\left(p^{2}-q^{2}\right)u^{2}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\int_{0}^{\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}}\mathrm{d}v\,\frac{q}{\sqrt{p^{2}-q^{2}}}\cdot\frac{1}{q^{2}+q^{2}v^{2}};~~~\small{\left[u=\frac{qv}{\sqrt{p^{2}-q^{2}}}\right]}\\ &~~~~~-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi}{2}\cdot\frac{1}{q\sqrt{p^{2}-q^{2}}}\int_{0}^{\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}}\mathrm{d}v\,\frac{1}{1+v^{2}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}.\tag{3a}\\ \end{align}$$
Then $0<-1+\sqrt{1+p^{2}}<p$, and
$$\begin{align} \mathcal{I}{\left(p,q\right)} &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{0}^{1}\mathrm{d}t\,\frac{\arctan{\left(\frac{1}{\sqrt{t^{2}+p^{2}}}\right)}}{\left(t^{2}+q^{2}\right)\sqrt{t^{2}+p^{2}}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\int_{p}^{-1+\sqrt{1+p^{2}}}\mathrm{d}x\,\frac{(-1)\left(p^{2}+x^{2}\right)}{2x^{2}}\cdot\frac{1}{\left(\frac{p^{2}-x^{2}}{2x}\right)^{2}+q^{2}}\cdot\frac{2x}{p^{2}+x^{2}}\\ &~~~~~\times\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)};~~~\small{\left[-t+\sqrt{t^{2}+p^{2}}=x\right]}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{-1+\sqrt{1+p^{2}}}^{p}\mathrm{d}x\,\frac{4x\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)}}{\left(p^{2}-x^{2}\right)^{2}+4q^{2}x^{2}}.\tag{3b}\\ \end{align}$$
It follows from the arctangent addition formula that
$$\arctan{\left(\frac{x}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{x}{1+\sqrt{1+p^{2}}}\right)}=\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)};~~~\small{p\in\mathbb{R}_{>0}\land x\in\mathbb{R}}.\tag{4}$$
Set $r:=\frac{p}{1+\sqrt{1+p^{2}}}\land\sigma:=\arcsin{\left(\frac{q}{p}\right)}$. Then, $0<r<1\land0<\sigma<\frac{\pi}{2}\land r^{-1}=\frac{1+\sqrt{1+p^{2}}}{p}=\frac{p}{-1+\sqrt{1+p^{2}}}$, and
$$\begin{align} \mathcal{I}{\left(p,q\right)} &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{-1+\sqrt{1+p^{2}}}^{p}\mathrm{d}x\,\frac{4x\arctan{\left(\frac{2x}{p^{2}+x^{2}}\right)}}{\left(p^{2}-x^{2}\right)^{2}+4q^{2}x^{2}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{-1+\sqrt{1+p^{2}}}^{p}\mathrm{d}x\,\frac{4x}{\left(p^{2}-x^{2}\right)^{2}+4q^{2}x^{2}}\\ &~~~~~\times\left[\arctan{\left(\frac{x}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{x}{1+\sqrt{1+p^{2}}}\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\int_{\frac{-1+\sqrt{1+p^{2}}}{p}}^{1}\mathrm{d}y\,\frac{4p^{2}y}{\left(p^{2}-p^{2}y^{2}\right)^{2}+4p^{2}q^{2}y^{2}}\\ &~~~~~\times\left[\arctan{\left(\frac{py}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{py}{1+\sqrt{1+p^{2}}}\right)}\right];~~~\small{\left[x=py\right]}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{\frac{-1+\sqrt{1+p^{2}}}{p}}^{1}\mathrm{d}y\,\frac{4y}{\left(1-y^{2}\right)^{2}+4\left(\frac{q}{p}\right)^{2}y^{2}}\\ &~~~~~\times\left[\arctan{\left(\frac{py}{-1+\sqrt{1+p^{2}}}\right)}-\arctan{\left(\frac{py}{1+\sqrt{1+p^{2}}}\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y}{\left(1-y^{2}\right)^{2}+4y^{2}\sin^{2}{\left(\sigma\right)}}\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]}{1-2y^{2}\left[1-2\sin^{2}{\left(\sigma\right)}\right]+y^{4}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]}{1-2y^{2}\cos{\left(2\sigma\right)}+y^{4}}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]}{\left[1-2y\cos{\left(\sigma\right)}+y^{2}\right]\left[1+2y\cos{\left(\sigma\right)}+y^{2}\right]}\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}\cos{\left(\sigma\right)}}\int_{r}^{1}\mathrm{d}y\,\frac{4y\cos{\left(\sigma\right)}}{\left[1-2y\cos{\left(\sigma\right)}+y^{2}\right]\left[1+2y\cos{\left(\sigma\right)}+y^{2}\right]}\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}\cos{\left(\sigma\right)}}\int_{r}^{1}\mathrm{d}y\,\left[\frac{1}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{1}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{p^{2}\sin{\left(\sigma\right)}\cos{\left(\sigma\right)}}\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\frac{\pi\arctan{\left(\frac{\sqrt{p^{2}-q^{2}}}{q\sqrt{1+p^{2}}}\right)}}{2q\sqrt{p^{2}-q^{2}}}\\ &~~~~~-\frac{1}{q\sqrt{p^{2}-q^{2}}}\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right].\tag{5}\\ \end{align}$$
As such, let's introduce another auxiliary function $\mathcal{J}:\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)\rightarrow\mathbb{R}$ defined via the last definite integral above:
$$\mathcal{J}{\left(r,\sigma\right)}:=\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right].\tag{6}$$
Given $\left(r,\sigma\right)\in\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)$, we find
$$\begin{align} \mathcal{J}{\left(r,\sigma\right)} &=\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\\ &~~~~~\times\left[\arctan{\left(\frac{y}{r}\right)}-\arctan{\left(ry\right)}\right]\\ &=\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\arctan{\left(\frac{y}{r}\right)}\\ &~~~~~-\int_{r}^{1}\mathrm{d}y\,\left[\frac{\sin{\left(\sigma\right)}}{1-2y\cos{\left(\sigma\right)}+y^{2}}-\frac{\sin{\left(\sigma\right)}}{1+2y\cos{\left(\sigma\right)}+y^{2}}\right]\arctan{\left(ry\right)}\\ &=\int_{1}^{\frac{1}{r}}\mathrm{d}t\,\left[\frac{r\sin{\left(\sigma\right)}}{1-2rt\cos{\left(\sigma\right)}+r^{2}t^{2}}-\frac{r\sin{\left(\sigma\right)}}{1+2rt\cos{\left(\sigma\right)}+r^{2}t^{2}}\right]\arctan{\left(t\right)};~~~\small{\left[y=rt\right]}\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)};~~~\small{\left[y=\frac{u}{r}\right]}\\ &=\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\\ &~~~~~\times\arctan{\left(\frac{1}{u}\right)};~~~\small{\left[t=u^{-1}\right]}\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\left[\frac{\pi}{2}-\arctan{\left(u\right)}\right]\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\\ &~~~~~-\int_{r}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &~~~~~-\int_{r^{2}}^{r}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\int_{r}^{1}\mathrm{d}u\,\frac{d}{du}\left[\arctan{\left(\frac{u-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{u+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\right]\\ &~~~~~-\int_{r^{2}}^{1}\mathrm{d}u\,\left[\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}-\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\right]\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\bigg{[}\arctan{\left(\frac{1-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{1+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &~~~~~-\arctan{\left(\frac{1-\cos{\left(\sigma\right)}}{\sin{\left(\sigma\right)}}\right)}+\arctan{\left(\frac{1+\cos{\left(\sigma\right)}}{\sin{\left(\sigma\right)}}\right)}\bigg{]}\\ &~~~~~+\int_{r^{2}}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &~~~~~-\int_{r^{2}}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}-2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &=\frac{\pi}{2}\left[\frac{\pi}{2}-\sigma-\arctan{\left(\frac{r^{2}\sin{\left(2\sigma\right)}}{1-r^{2}\cos{\left(2\sigma\right)}}\right)}\right]\\ &~~~~~+\int_{r^{2}}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &~~~~~+\int_{-1}^{-r^{2}}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)};~~~\small{\left[u\mapsto-u\right]}\\ &=\frac{\pi}{2}\left[\frac{\pi}{2}-\sigma-\arctan{\left(\frac{r^{2}\sin{\left(2\sigma\right)}}{1-r^{2}\cos{\left(2\sigma\right)}}\right)}\right]\\ &~~~~~+\int_{-1}^{1}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &~~~~~-\int_{-r^{2}}^{r^{2}}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}.\tag{7}\\ \end{align}$$
To facilitate the evaluation of the remaining two integrals in the last line above, we introduce yet another auxiliary function $\mathcal{K}:\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)\times\left(0,1\right]\rightarrow\mathbb{R}$ defined via the definite integral
$$\mathcal{K}{\left(r,\sigma,z\right)}:=\int_{-z}^{z}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}.\tag{8}$$
Suppose $\left(r,\sigma,z\right)\in\left(0,1\right)\times\left(0,\frac{\pi}{2}\right)\times\left(0,1\right]$. We then have
$$\begin{align} \mathcal{K}{\left(r,\sigma,z\right)} &=\int_{-z}^{z}\mathrm{d}u\,\frac{r\sin{\left(\sigma\right)}}{r^{2}+2ru\cos{\left(\sigma\right)}+u^{2}}\arctan{\left(u\right)}\\ &=\int_{-z}^{z}\mathrm{d}u\,\arctan{\left(u\right)}\frac{d}{du}\arctan{\left(\frac{u+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &=\arctan{\left(z\right)}\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(-z\right)}\arctan{\left(\frac{-z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &~~~~~-\int_{-z}^{z}\mathrm{d}u\,\arctan{\left(\frac{u+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\frac{d}{du}\arctan{\left(u\right)};~~~\small{I.B.P.}\\ &=\arctan{\left(z\right)}\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(z\right)}\arctan{\left(\frac{z-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\\ &~~~~~-\int_{-z}^{z}\mathrm{d}u\,\frac{1}{1+u^{2}}\arctan{\left(\frac{r\cos{\left(\sigma\right)}+u}{r\sin{\left(\sigma\right)}}\right)}\\ &=\arctan{\left(z\right)}\left[\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{z-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\right]\\ &~~~~~-\int_{-z}^{z}\mathrm{d}u\,\frac{1}{1+u^{2}}\arctan{\left(\cot{\left(\sigma\right)}+r^{-1}u\csc{\left(\sigma\right)}\right)}\\ &=\arctan{\left(z\right)}\left[\arctan{\left(\frac{z+r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}-\arctan{\left(\frac{z-r\cos{\left(\sigma\right)}}{r\sin{\left(\sigma\right)}}\right)}\right]\\ &~~~~~-\int_{-\arctan{\left(z\right)}}^{\arctan{\left(z\right)}}\mathrm{d}\varphi\,\arctan{\left(\cot{\left(\sigma\right)}+r^{-1}\csc{\left(\sigma\right)}\tan{\left(\varphi\right)}\right)};~~~\small{\left[\arctan{\left(u\right)}=\varphi\right]}.\tag{9}\\ \end{align}$$
Suppose $\left(a,b,\psi,\omega\right)\in\mathbb{R}\times\mathbb{R}\times\left(-\frac{\pi}{2},\frac{\pi}{2}\right)\times\left(-\frac{\pi}{2},\frac{\pi}{2}\right)$, and assume
$$0<a\land\sqrt{1+a^{2}}<b\land-\frac{\pi}{4}\le\psi<\omega\le\frac{\pi}{4}.$$
Set
$$B:=\frac{\left[\sqrt{a^{2}+\left(b+1\right)^{2}}+\sqrt{a^{2}+\left(b-1\right)^{2}}\right]^{2}}{4b}\land\phi:=\frac12\arctan{\left(\frac{2ab}{b^{2}-a^{2}-1}\right)},$$
and note that $1<B\land0<\phi<\frac{\pi}{4}$. Then, it can be shown that
$$\begin{align} \int_{\psi}^{\omega}\mathrm{d}\varphi\,\arctan{\left(a+b\tan{\left(\varphi\right)}\right)} &=\frac12\left(\phi+\omega\right)^{2}-\frac12\left(\phi+\psi\right)^{2}\\ &~~~~~-\left(\omega-\psi\right)\arctan{\left(\frac{\tan{\left(\phi\right)}}{B}\right)}\\ &~~~~~-\frac12\operatorname{Li}_{2}{\left(\frac{1-B}{1+B},\pi-2\phi-2\omega\right)}\\ &~~~~~+\frac12\operatorname{Li}_{2}{\left(\frac{1-B}{1+B},\pi-2\phi-2\psi\right)}.\tag{10}\\ \end{align}$$
The integration formula above is derived in this question.
Integration formula $(10)$ is sufficient to provide us with a closed-form expression for $\mathcal{K}$, and in turn $\mathcal{J}$ and $\mathcal{I}$ as well. So our work is complete in principle, and all that remains is to substitute back the chain of results to obtain a final expression in terms of the original variables. (Forgive me if I don't bother to do that here since the end result is such a cumbersome expression.)