Without loss of generality, suppose that $[XYZ]$ (the area of $\triangle XYZ$) is $1$, and the ratio of similarity between $\triangle DEF$ and $\triangle XYZ$ is $r<1$ (so that $[DEF] = r^2$).
Let $a, b, c$ be the distances between $EF$ and $YZ$, between $ZX$ and $FD$, and between $XY$ and $DE$, respectively.
Then we have $[AEF] = \frac a2 \cdot EF$, $[BFD] = \frac b2 \cdot FD$, and $[CDE] = \frac c2 \cdot DE$ by the formula for triangle area; adding them together, we have $$[ABC] - [DEF] = \frac a2 \cdot EF + \frac b2 \cdot FD + \frac c2 \cdot DE.$$
On the other hand, we have $[AEY] = \frac a2 \cdot AY$, $[AFZ] = \frac a2 \cdot AZ$, $[BFZ] = \frac b2 \cdot BZ$, $[BDX] = \frac b2 \cdot BX$, $[CDX] = \frac c2 \cdot CX$, and $[CEY] = \frac c2 \cdot CY$; adding them together and noting that for example $YZ = AY + AZ$, we have $$[XYZ] - [ABC] = \frac a2 \cdot YZ + \frac b2 \cdot ZX + \frac c2 \cdot XY.$$
Because $r$ is the ratio of similarity between $\triangle DEF$ and $\triangle XYZ$, we have $EF = r \cdot YZ$, $FD = r \cdot ZX$, and $DE = r \cdot XY$, which tells us that
$$
[ABC] - [DEF] = r([XYZ] - [ABC]).
$$
Recall that we assumed $[XYZ] = 1$ and $[DEF] = r^2$, so we now have $[ABC] - r^2 = r(1 - [ABC])$. Solving, we get $[ABC] = r$, so $[ABC] = \sqrt{r^2 \cdot 1} = \sqrt{[DEF] \cdot [XYZ]}$.
I'm not sure how formal a proof you want, but I'll show you what I came up with.
- Draw triangle $ABC$.
- WLOG, let $A'$ have the same position as $A$ and $D'$ the same position as $D$ (using $\ A'D' = AD$).
- Try different positions of $B'$ and $C'$ using: $1)\ A'B' = AB,\ 2)\ A'C' = AC$ and $3)\ \angle BAD:\angle CAD=\angle B'A'D':\angle C'A'D'.$
For example, Suppose $\angle B'A'D' < \angle BAD,\ $ that is, $\exists \gamma \in\ [0,1)$ such that $\angle B'A'D' = \gamma (\angle BAD)$. Then $\angle D'A'C' = \gamma (\angle DAC)$. But $D'$ is a fixed point (by assumption) and we see it is not on the line $B'C'$.
The green circle has centre $A$ and passes through $B$ and $B'$. The blue circle has centre $A$ and passes through $C$ and $C'$.
You can use a similar argument for $\angle B'A'D' > \angle BAD.$
Best Answer
First you should prove that $\triangle DEF \cong \triangle GHI$ by applying the ASA (Angle-Side-Angle) criterion:
angle $EDF \cong$ angle $HGI$;
$DF \cong GI$;
angle $DFE \cong$ angle $GIH$.
Finally you can prove that $\triangle ABC \cong \triangle GHI$ by applying the transitive property of congruence.