When you scale a polygon (or any subset of the plane whose area you can define) by $\alpha$, its area scales by $\alpha^2$. Proving this rigorously takes some work and requires you to have a rigorous definition of "area", but you can see that this makes sense by considering a rectangle. If you scale a rectangle by $\alpha$, you scale each of its sides by $\alpha$, so since the area is the product of the two side lengths, you scale the area by $\alpha^2$.
So in your case, you want $\alpha^2A=A'$, or $\alpha=\sqrt{A'/A}$.
Denote area of $\triangle ABC$ by $\Delta$. Let areas of the three quadrilaterals each be $X$ and the blue triangles in following diagram, each be $Y$.
We'll take ratios approach, which is fun, form two linear equations in $X,Y$ and solve for it.
First notice $\triangle$s $AED$, $BFE$, $CDF$ are all $30^\circ-60^\circ-90^\circ$. Each of these e.g., $\triangle BFE$ has base one-third and height two-third the original, hence area of equilateral $\triangle DFE$ is
$$[DFE] = \Delta \left(1-3\cdot \frac{1}{3} \cdot \frac{2}{3}\right)=\frac{\Delta}{3}$$
If you don't know the hatched triangle is the one-seventh area triangle, we can also activate Routh's theorem to obtain its area as $\Delta/7$. We write
$$[DFE] = \frac{\Delta}{3} = 3X + 3Y + \frac{\Delta}{7} \tag{1}$$
Next $\triangle AFE$ has same height but one-third base of $\triangle ABF$ which in turn has one-third base of $\triangle ABC$, so $[AFE] = \Delta/9$.
Drop $FP \perp AB$. $\triangle BPF$ is also $30^\circ-60^\circ-90^\circ$. Let $BF=a$. $\triangle AQE \sim \triangle AFP$ with ratio $AE/AP=AE/(AB-BP)=2/5$. Thus $AQE$ has same base but two-fifths the height of $\triangle AFE$. We can write
$$[AFE]=\frac{\Delta}{9}=X+2Y+\frac{2}{5}\cdot \frac{\Delta}{9} \tag{2}$$
On simplifying $(1),(2)$,
$$3X+3Y=\frac{4}{21}\Delta \tag{3}$$
$$X+2Y=\frac{1}{15}\Delta \tag{4}$$
Eliminating $Y$, we get
$$3X=\left(2\cdot \frac{4}{21} - 3\cdot \frac{1}{15} \right) \Delta$$
That is
$$\boxed{\text{Area of quadrilaterals} = \frac{19}{105}\Delta}$$
which is about $18 \%$ of area of $\triangle ABC$.
Best Answer
Let $O$ is the origin. Denote "signed area" of triangle $OAB$: $~~S_{OAB}= \dfrac{1}{2}(x_Ay_B-x_By_A)$.
It can be derived from cross product of vectors $\vec{OA}, \vec{OB}$.
If way $AB$ is $\circlearrowleft$ (if polar angle of $A$ less than polar angle of $B$), then $S_{OAB}>0$ ;
if way $AB$ is $\circlearrowright$ (if polar angle of $A$ greater than polar angle of $B$), then $S_{OAB}<0$.
Now, for each edge $A_jA_{j+1}$ ($j=1,2,\ldots,n$; $A_{n+1}\equiv A_1$) of polygon $A_1A_2\ldots A_n$ we can build $2$ vectors: $\vec{OA_j}$ and $\vec{OA_{j+1}}$.
And "signed area" of polygon (which sign depends on numerating direction) $$ S_{A_1A_2...A_n} = \sum_{j=1}^n S_{OA_jA_{j+1}} = \sum_{j=1}^n \dfrac{1}{2}(x_jy_{j+1}-x_{j+1}y_j) = \dfrac{1}{2}\sum_{j=1}^n (x_jy_{j+1}-x_{j+1}y_j). $$
When positive term adds, then square will increase, when negative, then area will decrease.
We will mark "positive" area as blue, and "negative" as red.
Illustration:
