Instead of doing the $u$ substitution before evaluating the integral in $v$, it's easier to do it after computing that integral. Here is a detailed computation.
(I've assumed that you want to evaluate this integral and not simply to show that it is equal to $\zeta(2)$. This exercise appears in several references: Tom Apostol's A Proof that Euler Missed: Evaluating $\zeta(2)$ the Easy Way, Martin Aigner and Günter Ziegler's Proofs from The BOOK and as an exercise in a number theory text by LeVeque .)
By the substitution $x=\frac{\sqrt{2}}{2}\left( u-v\right) ,y=\frac{\sqrt{2}}{2}\left( u+v\right) $, whose Jacobian $J=\frac{\partial (x,y)}{\partial
(u,v)}=1$, the region of integration becomes the blue square in the $u,v$-plane with vertices
$$(0,0),\left(\frac{\sqrt{2}}{2},-\frac{\sqrt{2}}{2}\right),(\sqrt{2},0),\left(\frac{\sqrt{2}}{2},\frac{\sqrt{2}}{2}\right), $$ as shown in the following figure.
Observing that
$$
\frac{1}{1-xy}=\frac{2}{2-u^2+v^2}
$$
is symmetric in $v$, we get
$$\begin{eqnarray*}
I &=&\int_0^1\int_0^1 \frac{1}{1-xy}\,dx\,dy & x=\frac{\sqrt{2}}{2} (u-v) ,\quad y=\frac{\sqrt{2}}{2} (u+v) \\
&=&2\int_{u=0}^{\sqrt{2}/2}\int_{v=0}^{u} \frac{2}{2-u^{2}+v^{2}}\times
1\,du\,dv \\
&&{}+2\int_{u=\sqrt{2}/2}^{\sqrt{2}}\int_{v=0}^{\sqrt{2}-u}\frac{2}{
2-u^2+v^2}\times 1\,du\,dv \\
&=&4\int_0^{\sqrt{2}/2}\left( \int_{0}^{u}\frac{dv}{2-u^{2}+v^{2}}\right)
\,du\, \\
&&{}+4\int_{\sqrt{2}/2}^{\sqrt{2}}\left( \int_{0}^{\sqrt{2}-u}\frac{dv}{
2-u^{2}+v^{2}}\right) \,du \\
&=&4\int_{0}^{\sqrt{2}/2}\frac{1}{\sqrt{2-u^{2}}}\arctan \frac{u}{\sqrt{
2-u^{2}}}\,du, & u=\sqrt{2}\sin t \\
&&{}+4\int_{\sqrt{2}/2}^{\sqrt{2}}\frac{1}{\sqrt{2-u^{2}}}\arctan \frac{\sqrt{2
}-u}{\sqrt{2-u^{2}}}\,du\, & u=\sqrt{2}\cos \theta \\
&=&4\int_{0}^{\pi /6}\arctan(\tan t)\, dt\\&&{}+4\int_0^{\pi
/3}\arctan \left( \frac{1-\cos \theta}{\sin \theta}\right) d\theta, \\
&=&4\int_{0}^{\pi /6}t\,dt+4\int_0^{\pi /3}\arctan \left( \tan \frac{\theta}{2} \right) \, d\theta \\
&=&\frac{\pi^2}{18}+\frac{\pi^2}{9}=\frac{\pi^2}{6}.
\end{eqnarray*}
$$
One of the substitutions is a new one. After the first pair of substitutions $x,y$, we have done two additional ones in the resulting integrals, as indicated above: in the 1st, $u=\sqrt{2}\sin t,du=\sqrt{2}\cos t\,dt$, and in the 2nd, $u=\sqrt{2}\cos \theta,du=-\sqrt{2}\sin \theta\,d\theta$.
You are not using the correct bounds for the integral as correctly mentioned in the other answer.
The region of interest is $$R=\{(x,y): x\ge 0, y\ge 0, x+y\le 1\}$$
A different change of variables, namely $$(x,y)\to (u,v)\text{ such that } u=\frac{x-y}{x+y}, v=x+y$$
makes the new ranges independent of each other. Since the region $R$ is transformed to some $$D=\{(u,v): -1\le u\le 1, 0\le v\le 1\}$$
The absolute value of the Jacobian is $v/2$, so that the integral is
\begin{align}
I=&\iint_D \cos u \left(\frac{v}{2}\right)\, du\, dv
\\&=\frac{1}{2}\int_{-1}^{1} \cos u\,du\int_0^1 v\,dv
\end{align}
Best Answer