I'd like to find a way to rigorously evaluate the limit
$$\lim\limits_{t\to 1^-}\int_{\sin^{-1}\left(\frac{t}{2}\right)}^{\sin^{-1}(t)}\tan^{-1}\left(\frac{t – \sin x}{\cos x}\right)dx$$
This limit arises when one attempts to solve the Basel problem by evaluating the integral
$$I(t)=\int_0^t\int_0^t\frac{1}{1-xy}dydx$$
using two methods: (1) expanding the integrand as a geometric series, and (2) using the change of variables $x=(u-v)/\sqrt2$, $y=(u+v)/\sqrt2$. Doing so, we find that for every $t\in(-1,1)$,
$$I(t)=\sum_{n=1}^{\infty}\frac{t^{2n}}{n^2}$$
$$\text{ and }$$
$$I(t)=2\left(\sin^{-1}\frac{t}{2}\right)^2+4\int_{\sin^{-1}\left(\frac{t}{2}\right)}^{\sin^{-1}(t)}\tan^{-1}\left(\frac{t – \sin x}{\cos x}\right)dx$$
Now, the series $\sum t^{2n}/n^2$ evaluated at $t=1$ is $\sum 1/n^2$. This series is convergent, so Abel's theorem implies that $I$ is continuous from the left at $1$, and that $I(1)$ is the sum of the series $\sum 1/n^2$ (a similar conclusion can be reached for the value $-1$). Since $2\left(\sin^{-1} t/2\right)^2$ is continuous on $[-1,1]$, it follows that the integral
$$\int_{\sin^{-1}\left(\frac{t}{2}\right)}^{\sin^{-1}(t)}\tan^{-1}\left(\frac{t – \sin x}{\cos x}\right)dx = \frac{1}{4}\left(I(t)-2\left(\sin^{-1}\frac{t}{2}\right)^2\right)$$
is continuous on $[-1,1]$, and that we should be able to compute
$$\lim\limits_{t\to 1^-}\int_{\sin^{-1}\left(\frac{t}{2}\right)}^{\sin^{-1}(t)}\tan^{-1}\left(\frac{t – \sin x}{\cos x}\right)dx$$
by directly substituting in the value $t=1$.
I'm a bit skeptical about the validity of this reasoning, though. What makes me uncomfortable is the fact that the integrand of the resulting integral
$$\int_{\frac{\pi}{6}}^{\frac{\pi}{2}}\tan^{-1}\left(\frac{1-\sin x}{\cos x}\right)dx$$
has a singularity at $\pi/2$, an element of the domain of integration, so the integrand has suddenly gone from being free of singularities to having one. Granted, the singularity in question is removable, but the fact that singularities aren't present for $t\in(-1,1)$ and that $I$ is continuous at $1$ really makes me wonder "Why does the integrand suddenly accrue a singularity at $t=1$?". Can someone reassure me that this procedure is valid, and, if you can, explain what's going on for $t=1$? If it's not valid, how can I rigorously evaluate the limit in question?
Edit I should've mentioned this in the original post. Heuristically, I know that the limit is exactly $\pi^2/36$. This follows from "evaluating" the limiting integral as follows:
\begin{align*}
\int_{\frac{\pi}{6}}^{\frac{\pi}{2}}\tan^{-1}\left(\frac{1-\sin x}{\cos x}\right)dx &= \int_{\frac{\pi}{6}}^{\frac{\pi}{2}}\tan^{-1}\left(\frac{1-\cos\left(\frac{\pi}{2}-x\right)}{\sin\left(\frac{\pi}{2}-x\right)}\right)dx\\
&= \int_{\frac{\pi}{6}}^{\frac{\pi}{2}}\tan^{-1}\left(\tan \frac{\frac{\pi}{2}-x}{2}\right)dx\\
&= \int_{\frac{\pi}{6}}^{\frac{\pi}{2}}\left(\frac{\pi}{4}-\frac{x}{2}\right)dx\\
&= \frac{\pi^2}{36}
\end{align*}
What I'm more interested in is learning a way to establish this limit rigorously, provided that this integral method does not suffice. I apologize for any confusion.
Best Answer
Surprising or not, for
$$I(t)=\int_{\sin^{-1}\left(\frac{t}{2}\right)}^{\sin^{-1}(t)}\tan^{-1}\left(\frac{t - \sin (x)}{\cos (x)}\right)dx$$ a CAS provide an analytical result (the result would require five or six pages to be printed). Have a look here for the antiderivative.
The problem is that my computer almost died when I asked for the limit when $t\to 1^-$. Even worse trying a series expansion.
So ,let $t=1-\epsilon$ and consider $$I(\epsilon)=\int^{\sin ^{-1}(1-\epsilon ) }_{\sin ^{-1}\left(\frac{1-\epsilon }{2}\right)} \tan ^{-1}\Big[\sec (x) ((1-\epsilon)- \sin (x))\Big]\,dx$$
Developing the integrand as a series around $\epsilon=0$ leads to a sum of rather simpler trigonometric integrals. Use the bounds and, again, expand the result a series around $\epsilon=0$. This leads to $$I(\epsilon)=\sum_{n=0}^p a_n\,\epsilon^n +O(\epsilon^{p+1})$$
The very first coefficients are $$a_0=\frac{\pi ^2}{36}$$ $$a_1=\frac{\log (2\epsilon)}2+ \frac{\sqrt{3} \pi -9}{18} $$ $$a_2=\frac{\log (2\epsilon)}4-\frac{\sqrt{3} \pi +45}{108} $$ $$a_3=\frac{\log (2\epsilon)}6+\frac{8 \sqrt{3} \pi -135}{1296} $$ $$a_4=\frac{\log (2\epsilon)}8-\frac{196 \sqrt{3} \pi +8667}{54432} $$ $$a_5=\frac{\log (2\epsilon)}{10}+\frac{2660 \sqrt{3} \pi -16767} { 1020600}$$
Using the above and comparing with high precision numerical integration gives for the decimal logarithm of the absolute difference the following results (with $\epsilon=10^{-k}$) $$\left( \begin{array}{cc} k & \log_{10}(|\Delta|) \\ 1 & -6.4016 \\ 2 & -10.603 \\ 3 & -14.533 \\ 4 & -18.527 \\ 5 & -22.526 \\ 6 & -26.526 \\ 7 & -30.526 \\ 8 & -34.526 \\ 9 & -38.526 \\ 10 & -42.526 \end{array} \right)$$ which is almost $$\log_{10}(|\Delta|) \sim -2.5-4.0 \,k$$
Edit
Consider $$\int\tan^{-1}\left(\frac{1-\sin (x)}{\cos (x)}\right)dx=\frac{1}{4} x \left(x+4 \tan ^{-1}(\sec (x)-\tan (x))\right)$$ $$\int_{\frac{\pi}{6}}^{\frac{\pi}{2}-\epsilon}\tan^{-1}\left(\frac{1-\sin (x)}{\cos (x)}\right)dx=\frac{1}{16} (\pi -2 \epsilon ) \left(-2 \epsilon +8 \tan ^{-1}\left(\tan \left(\frac{\epsilon }{2}\right)\right)+\pi \right)-\frac{5 \pi ^2}{144}$$ Expand as series to obtain exactly $$\frac{\pi ^2}{36}-\frac{\epsilon ^2}{4}$$
For $\epsilon=10^{-5}$, this would give a difference of $1.12 \times 10^{-16}$.