We are given:
$$ y = \int\frac{\sec x\,dx}{\sqrt{3+\tan x}}. $$
First, change the variable of integration from $x$ to $u=\sqrt{3+\tan x}$; we have that
\begin{align*}
du &= \frac{\sec^{2}x\,dx}{2\sqrt{3+\tan x}} \\
\therefore dy &= \frac{2\,du}{\sec x} \\
&= \frac{2\,du}{\sqrt{1+(u^{2}-3)^{2}}}.
\end{align*}
Now, if there exists $a$ with $u=a t$ such that
$$ 1 + (u^{2}-3)^{2} = r^{2}(1-t^{2})(1-k^{2}t^{2}) $$
for some $r,k$, then changing the variable of integration from $u$ to $t$ turns our integral into an incomplete elliptic one of the first kind.
To find such $a,r,k$, expand the powers and rewrite $u$ as $at$ in the previous equation to yield
$$a^{4}t^{4}-6a^{2}t^{2}+10=r^{2}k^{2}t^{4}-r^{2}(1+k^{2})t^{2}+r^{2}.$$
Equating coefficients of like powers immediately yields
\begin{align*}
r^{2}=10 && 6a^{2}=r^{2}(1+k^{2}) && a^{4}=r^{2}k^{2}
\end{align*}
which gives us (but not before some sacrifice)
\begin{align*}
k=\frac{3\pm i}{\sqrt{10}}&&
k^{2}=\frac{4\pm 3i}{5}&&
a=\sqrt{3\pm i}.
\end{align*}
So now return to our integral
$$ y = 2\int\frac{du}{\sqrt{1+(u^{2}-3)^{2}}} $$
and substitute $u$ for $t$, whereby it becomes
\begin{align*}
y &= 2a\int\frac{dt}{\sqrt{r^{2}(1-t^{2})(1-k^{2}t^{2})}}\\
&=\frac{2a}{r}\int\frac{dt}{\sqrt{(1-t^{2})(1-k^{2}t^{2})}}\\
&=\frac{2a}{r}\mathrm{F}(t;k)+C.
\end{align*}
Finally, we undo our subsitutions,
\begin{align*}
t &= \frac{\sqrt{3+\tan x}}{a} \\
&= \sqrt{\frac{3\mp i}{10}}\sqrt{3+\tan x}
\end{align*}
expand
$$ \frac{2a}{r} = \sqrt{\frac{6\pm2i}{5}}, $$
and pass to Legendre's notation, to yield
$$ y=\int\frac{\sec x\,dx}{\sqrt{3+\tan x}}=\sqrt{\frac{6\pm2i}{5}}\mathrm{F}\left(\arcsin\left(\sqrt{\frac{3\mp i}{10}}\sqrt{3+\tan x}\right)\Bigg|\frac{4\pm3i}{5}\right)+C. \tag{$\star$}\label{star}$$
I have gone over the above a handful of times now, and can't find any mistakes.
One thing concerned me though: this is similar enough to David G. Stork's provided answer for me to believe it is not entirely incorrect, but at the same time, it is different enough for me to suspect it is not entirely correct either.
If one examines the differences however, one finds that
$$ \cos x \sqrt{(1+3i)+(3-i)\tan x}\sqrt{(-3-i)(\tan x +i)} = \pm(1+3i)$$
is locally constant everywhere it is defined. Thus \eqref{star} is at least everywhere a constant multiple of the other answer.
This behaviour is probably due to the participation of square-root and tangent in the integrand and whatnot, though I don't know enough complex analysis. Just mind any possible singularities in your path of integration...
Best Answer
To evaluate $\int \frac{2x}{(1-x^2)\sqrt{x^4-1}}dx$
$\Rightarrow\int \frac{-2x}{(x^2-1)^{\frac{3}{2}}\sqrt{x^2+1}}dx$
Now, one substitution that works but is rather tricky to find is,
Let $u=\sqrt{\frac{x^2+1}{x^2-1}}$
$\Rightarrow \frac{1}{2}(\frac{x^2-1}{x^2+1})^{\frac{1}{2}} \frac{(x^2-1)2x-(x^2+1)2x}{(x^2-1)^2}dx=du$
$\Rightarrow (\frac{x^2-1}{x^2+1})^{\frac{1}{2}} \frac{-2x}{(x^2-1)^2}dx=du$
$\Rightarrow \frac{-2x}{(x^2-1)^{\frac{3}{2}}\sqrt{x^2+1}}dx=du$
The integral reduces to
$\int du=u+c=\sqrt{\frac{x^2+1}{x^2-1}}+c$
Thus $\int \frac{2x}{(1-x^2)\sqrt{x^4-1}}dx=\sqrt{\frac{x^2+1}{x^2-1}}+c$