We consider the general case shown in $\bf{Fig. 1}$. Use the triangle $OQC$ to obtain $OC$ and $OA$.
$$OC=\frac{\sqrt{4 r^2-h^2}}{2},\space\space\space\space\therefore\space\space OA=AC-OC=s+h-\frac{\sqrt{4 r^2-h^2}}{2}.$$
Apply Pythagorean theorem to the triangle $OPA$ to express $s$ in terms of $r$ and $h$.
$$r^2=\left(\frac{s}{2}\right)^2+\left(s+h-\frac{\sqrt{4r^2-h^2}}{2}\right)\space\to\space s=\frac{4\sqrt{4 r^2-h^2}-3 h}{5} $$
The required area is given by,
$$A(S_1+S_2)=h^2+s^2=\frac{2}{25}\left(32 r^2+9 h^2-12 h\sqrt{4 r^2-h^2}\right)\tag{1}$$.
Now, we go on to find the minima ans maxima of (1) by equating its first derivative to zero.
$$\frac{d}{dh}\left(A\left(S_1+S_2\right)\right)=3 h\sqrt{4 r^2-h^2}-2\left(4 r^2-h^2 \right)+2 h^2=0$$
To make thing easy, we let $y=\sqrt{4 r^2-h^2}$ and obtain,
$$3 h y-2 y^2+2 h^2=\left(2 h-y\right)\left(h+2 y\right)=0.$$
From this, it is obvious that $y$ has only one acceptable value, i.e.
$$h=\frac{y}{2}=\frac{\sqrt{4 r^2-h^2}}{2}.$$
This can be solved to find the $y$ value, for which the $A\left(S_1+S_2\right)$ has an extremum.
$$h=\frac{2}{\sqrt{5}}r$$
Now we need the second derivative of (1) to determine what kind of extremum this is. The 2nd derivative is,
$$8h\sqrt{4 r^2-h^2}+12 r^2-6h^2.\tag{2}$$
When we substitute $\frac{2}{\sqrt{5}}r$ for $h$ in (2), we get,
$$\frac{64}{5}r^2+12 r^2-\frac{24}{5}r^2=20 r^2\ge 0.$$
Therefore, the value of $h$ we obtained above corresponds to a minimum of $A\left(S_1+S_2\right)$ and not to a maximum. This is most unfortunate, but then that is how life works. Now we have to show that (1) is monotonic. We do this by more arguing than calculating. But first, we need to know the minimum and maximum possible values of $h$. According to $\bf{Fig. 3}$ and $\bf{Fig. 2}$, maximum and minimum possible values of $h$ are $\sqrt{2}r$ and $\frac{2}{\sqrt{5}}r$ respectively.
We have seen that the minimum value of $A\left(S_1+S_2\right)$ corresponds to the minimum value of $h$. As per this and in the absence of any other local extrema, the value of $A\left(S_1+S_2\right)$, as a function, increases monotonically without an upper bound as the value of $h$ increases. However, $h$ has an upper bound equal to $\sqrt{2}r$. This also means that the maximum value of $A\left(S_1+S_2\right)$, as an area, occurs when $h$ has its maximum possible value.
$$\therefore\space\space\space A\left(S_1+S_2\right)_{\rm max}=\frac{52}{25}r^2\approx 2.08 r^2.$$
Denote area of $\triangle ABC$ by $\Delta$. Let areas of the three quadrilaterals each be $X$ and the blue triangles in following diagram, each be $Y$.
We'll take ratios approach, which is fun, form two linear equations in $X,Y$ and solve for it.
First notice $\triangle$s $AED$, $BFE$, $CDF$ are all $30^\circ-60^\circ-90^\circ$. Each of these e.g., $\triangle BFE$ has base one-third and height two-third the original, hence area of equilateral $\triangle DFE$ is
$$[DFE] = \Delta \left(1-3\cdot \frac{1}{3} \cdot \frac{2}{3}\right)=\frac{\Delta}{3}$$
If you don't know the hatched triangle is the one-seventh area triangle, we can also activate Routh's theorem to obtain its area as $\Delta/7$. We write
$$[DFE] = \frac{\Delta}{3} = 3X + 3Y + \frac{\Delta}{7} \tag{1}$$
Next $\triangle AFE$ has same height but one-third base of $\triangle ABF$ which in turn has one-third base of $\triangle ABC$, so $[AFE] = \Delta/9$.
Drop $FP \perp AB$. $\triangle BPF$ is also $30^\circ-60^\circ-90^\circ$. Let $BF=a$. $\triangle AQE \sim \triangle AFP$ with ratio $AE/AP=AE/(AB-BP)=2/5$. Thus $AQE$ has same base but two-fifths the height of $\triangle AFE$. We can write
$$[AFE]=\frac{\Delta}{9}=X+2Y+\frac{2}{5}\cdot \frac{\Delta}{9} \tag{2}$$
On simplifying $(1),(2)$,
$$3X+3Y=\frac{4}{21}\Delta \tag{3}$$
$$X+2Y=\frac{1}{15}\Delta \tag{4}$$
Eliminating $Y$, we get
$$3X=\left(2\cdot \frac{4}{21} - 3\cdot \frac{1}{15} \right) \Delta$$
That is
$$\boxed{\text{Area of quadrilaterals} = \frac{19}{105}\Delta}$$
which is about $18 \%$ of area of $\triangle ABC$.
Best Answer
Let $a=BC$, $b=AC$, $c=AB$ be the side lengths in the given triangle. Since $c^2=a^2+b^2$ we need two equations in $a,b$ to solve the issue.
$(1)$ First of all, there are similar triangles, the given one with proportion $CB:CA=a:b$, the one with corresponding proportion $(a-21):21$, and the one with corresponding proportion $21:(b-21)$, so $$ \frac ab = \frac{a-21}{21}= \frac{21}{b-21} \ . $$ Equivalently, $$ ab = 21(a+b)\ . $$
$(2)$ Consider now the other square, it has size $s=\sqrt{440}$. We have again two similar triangles in the picture, both have in $C$ the right angle, the given one with hypotenuse $c$ and height $h=ab/c$, and its "cut version" with hypotenuse $s$ and height $h-s$, so we obtain a second equation: $$ \frac {h-s}s = \frac hc\ . $$ Equivalently, $hc-sc=hs$, i.e. $hc^2=sc^2+hcs$, i.e. $abc=s(a^2+b^2+ab)$, and after squaring: $$ a^2b^2(a^2+b^2) = 440(a^2+b^2+ab)^2\ . $$ We solve the system involving the two equations, let $S$ be the sum $S=a+b$, the number asked for in the OP, and $P$ the product $P=ab$. Then we have: $$ \left\{ \begin{aligned} P &= 21 S\ , \\ P^2(S^2-2P) &= 440(S^2-P)^2\ . \end{aligned} \right. $$ We plug in $P$ from the first equation in the second equation, get: $$ (S + 420)(S - 462)S^2\ . $$ We have thus $\color{blue}{S=462}$, since we must reject for geometrical reasons the vanishing of the other factors.
To obtain $a,b$ explicitly, we have to solve the equation $x^2-Sx+P=0$, which is $x^2 - 462x + 21\cdot 462=0$. And indeed, the solutions $231\pm 63\sqrt{11}$ are real positive numbers, the two values of $a,b$.
As a final note, the given numbers are so ugly, that checking is done hard, hope there is no computational error in between.