There is a famous book of Polya ("How to solve it"), in which the problem of inscribing a square in a triangle is treated in a really interesting way, I strongly suggest the reading.
The inscribed square is clearly unique once we choose the triangle side where two vertices of the square lie. If we suppose that the square has two vertices on $AB$ and side $l$, then:
$$l+l\cot A + l\cot B = c,$$
so:
$$ l = \frac{c}{1+\cot A+\cot B} = \frac{2R \sin A \sin B \sin C}{\sin C + \sin A\sin B}=\frac{abc}{2Rc+ab},$$
where $R$ is the circumradius of $ABC$. In order to maximize $l$, you only need to minimize $2Rc+ab = 2R\left(c+\frac{2\Delta}{c}\right)$, or "land" the square on the side whose length is as close as possible to $\sqrt{2\Delta}$, where $\Delta$ is the area of $ABC$.
In such issues, you can "deconvolute" the difficulties, first of all by using the power of point $A$ with respect to circle (C)
in 2 different ways :
$$P(A,(C))=AE^2-r^2=PI^2 \tag{1}$$
Knowing that $AE=AD-ED=\sqrt{2}(8-r).$
(1) gives :
$$2(8-r)^2-r^2=64$$
This quadratic equation has the unique compatible solution
$$r=8(2-\sqrt{3})\tag{2}$$ (as you have obtained).
Edit : Here is the end of the computation.
Let us compute the tangent of the two angles at the base of triangle $ABJ$ :
$$\tan(\angle BAJ)=\tan(\alpha)=\tan(\frac{\pi}{4}+\arctan \frac{r}{8})=\frac{1+\frac{r}{8}}{1-\frac{r}{8}}=\frac{8+r}{8-r}\tag{1}$$
$$\tan(\angle JBA)=\tan(\frac{\pi}{2}-2\beta)=\cot(2 \beta)=\frac{1}{\tan(2 \beta)}=\frac{1- (\tan \beta)^2}{2 \tan \beta}=\frac{8(4-r)}{r(8-r)},\tag{3}$$
the last equality being a consequence of relationship :
$$\tan(\beta)=\frac{r}{8-r}\tag{3'}$$
If we denote by $P$ the foot of the altitude issued from $J$, and $h=JP$, we can write :
$$\begin{cases}AP&=&\frac{h}{\tan(\angle BJA)}\\
PB&=&\frac{h}{\tan(\angle JBA)}\end{cases}$$
adding them, we get, by dividing by $h$ :
$$\frac{8}{h}=\frac{r(8-r)}{8(4-r)}+\frac{8-r}{8+r}$$
from which (using (2))
$$h=\frac{8}{11}(6+\sqrt{3})\approx 5.623309678$$
is obtained and from there the area :
$$\frac{32}{11}(6+\sqrt{3})$$
of triangle $JAB$.
Best Answer
Hint: Observe that $5 = 1^2 + 2^2, 10 = 1^2+3^2$ and $13 = 2^2 + 3^2,$