Let the triangle be $ABC$. I think we may take for granted that an optimal rectangle must have all its corners on the side of the triangle, so two of them must lie on a single side, which we may take to be $AB$. Picture this with $AB$ horizontal and $C$ on top.
Now fold the triangle with one corner at $C$ and its opposite side being the top side of the rectangle across the rectangle, with $C$ ending up at a new position $C'$. Fold the other two small triangles, with corners at $A$ and $B$ respectively, similarly over the left and right edges of the rectangle. Note carefully that the sides of the folded top triangle align with the slanted sides of the other two folded triangles.
If $C'$ lies above $AB$, then the other two triangles will overlap, and all three will cover the rectangle, so the combined area of the small triangles is greater than the area of the rectangle.
If $C'$ lies below $AB$, the small triangles will cover the rectangle with a bit sticking outside, so again, theire combined area is greater than that of the rectangle.
If $C'$ lies on $AB$, the triangles will exactly cover the rectangle. This is the optimum.
Let the radius be $r$. The side lengths of the rectangle are $r-1$ and $r-2$. The diagonal is $r$. By the Pythagorean theorem we have
$$(r-1)^2+(r-2)^2=r^2$$
Which expands and simplifies to
$$r^2-6r + 5 = (r-5)(r-1) = 0$$
So $r=1$, or $r=5$. But, looking at the diagram again we see that the radius must be at least $2$, so $r=5$ is the answer.
Best Answer
Choosing one vertex to be $(r \cos\theta, r\sin\theta)$ then another is $(r (\cos\theta- \sin\theta),0)$, a third is $(0,r (\cos\theta- \sin\theta))$ and the fourth is $(r \sin\theta, r\cos\theta)$
The area is then $2r^2 \sin\theta(\cos\theta- \sin\theta) = \sqrt{8}r^2 \sin(\theta) \sin\left(\frac\pi4-\theta\right) $ which is maximised when $\theta= \frac\pi8$ with the maximum area being $r^2(\sqrt{2}-1)$
Added to illustrate comment: If the arrangement is not symmetric, you will not get two vertices on the arc and two on the axes. Instead you will get coomething like this