Let's call the circuit in the origin circuit one and it's line element $\mathrm{d}l_1=(0,\mathrm{d}y_1,0)$ and the one to it's right $r_2=(d,y_2,0)$ then the force between them is $\mathrm{d}F_{12}=i \mathrm{d}l_2 \times B_1$ where
$$B_1=\frac{\mu_0i}{4\pi}\int_{l_1} \frac{\mathrm{d}l_1\times \Delta r}{(\Delta r)^3}$$
and $\Delta r=(d,(y_2-y_1),0)$ so we have that
$$\mathrm{d}l_1\times \Delta r=(0,0,-\mathrm{d}y_1 d)$$
so we get as you wrote:
$$B_1=-\frac{\mu_0 i d}{4 \pi}\int_{0}^{d} \frac{\mathrm{d}y_1}{(d^2+(y_2-y_1)^2)^{\frac{3}{2}}}$$
Ok now let's call $y_2-y_1=t$ so $\mathrm{d}t=-\mathrm{d}y_1$ then we can write
$$B_1=\frac{\mu_0 i d}{4\pi}\int_{y_2}^{y_2-d}\frac{\mathrm{d}t}{(d^2+t^2)^{\frac{3}{2}}}$$
we now make the substitution
$$t=d\cdot \sinh(u)$$
and we obtain
$$\mathrm{d}t=d\cdot \cosh(u)\mathrm{d}u$$
and then
$$B_1=\frac{\mu_0 i d}{4\pi}\int \mathrm{d}u \frac{d \cosh(u)}{d^3 \cosh(u)^3}$$
in which we used
$$\cosh(u)^2-\sinh(u)^2=1$$
$$B_1=\frac{\mu_0 i }{4\pi d}\int \frac{\mathrm{d}u}{\cosh(u)^2}$$
now $\frac{1}{\cosh^2(u)}$ is the derivative of $\tanh(u)$ so
$$\int \frac{\mathrm{d}u}{\cosh(u)^2}=\tanh(u)$$
we get then
$$B_1=\frac{\mu_0 i }{4\pi d}\tanh\left(a\sinh\left(\frac{y_2-y_1}{d}\right)\right)+\text{const}$$
where we have substituted back all parameters
$$u=a\sinh\left(\frac{t}{d}\right) \\ t=y_2-y_1$$
so knowing that (where $a\sinh(x)$ is the inverse function of $\sinh(x)$):
$$\tanh(a\sinh(x))=\frac{x}{\sqrt{x^2+1}}$$
finally
$$B_1=\frac{\mu_0 i }{4\pi d} \frac{\frac{y_2-y_1}{d}}{\sqrt{(\frac{y_2-y_1}{d})^2+1}}+\text{const}=\frac{\mu_0 i }{4\pi} \frac{1}{\sqrt{(y_2-y_1)^2+d^2}}+\text{const}$$
now we calculate it between $y_1=0$ and $y_1=d$ which yields
$$B_1=\frac{\mu_0 i }{4\pi} \left[ \frac{1}{\sqrt{(y_2-d)^2+d^2}}-\frac{1}{\sqrt{y_2^2+d^2}}\right]$$
to calculate the force we take $B_1=(0,0,B_1 \hat{z})$ and we operate the following:
$$\mathrm{d}F_{12}=i\mathrm{d}l_2 \times B_1=i(B_1\mathrm{d}y_2,0,0)$$
now we have to integrate on the circuit two:
$$F_{12}=\frac{\mu_0 i^2 }{4\pi} \int_{0}^{d}\mathrm{d}y_2\left[ \frac{1}{\sqrt{(y_2-d)^2+d^2}}-\frac{1}{\sqrt{y_2^2+d^2}}\right]=\frac{\mu_0 i^2 }{4\pi} \left[I(y_2-d)-I(y_2)\right]$$
and know we do the same trick as before
$$t=y_2-d \ \text{or}\ t=y_2\ \text{for the second piece}$$
$$t=d\cdot \sinh(u)$$
$$\mathrm{d}y_2=\mathrm{d}t=d\cdot \cosh(u)\mathrm{d}u$$
then:
$$I=\int \mathrm{d}u\cdot d \cdot \cosh(u) \frac{1}{\sqrt{d^2\cosh^2(u)}}=u=a\sinh\left(\frac{t}{d}\right)$$
we finally get
$$F_{12}=\frac{\mu_0 i^2 }{4\pi}\left[a\sinh\left(\frac{y_2-d}{d}\right)-a\sinh\left(\frac{y_2}{d}\right)\right]_0^d$$
which curiously enough is zero for this choice of parameters! I hope that helped!
Best Answer
I'm not sure I understand how the loop is set but your case must be similar to the left and right side of the loop below.Since the current is flowing in a loop, on the left side it will flow the opposite direction of the way it does on the other side.Using the right-hand rule you will now see that the forces are equal and opposite,as shown in the picture, explaining the minus in your problem.
So, watch out which direction the current is going, always use the R.H.R. and carefully you will have no problem.