The following problem is taken from a Swedish 12th grade ‘Student Exam’ from 1934.
$\mathit{ABCD}$ is a rectangle with the sides $\mathit{AB}=15$ cm and $\mathit{BC}=12$ cm. On $\mathit{AB}$, between $A$ and $B$, is a point $E$ placed such that $\mathit{AE}=2$ cm. There are two rectangles, inscribed in the given rectangle, which has a corner at $E$ and corners on each of the three other sides of the given rectangle. Calculate the area of the two rectangles.
Below I present two solutions which have the correct answer in two different ‘formats’. The latter is the one given as answer in a book. The first solution, that gives the more ‘convoluted’ format, also includes an equation that is, perhaps, not as easy to solve unless one observes a particular (imaginary) solution first. Also, the calculations are pretty cumbersome to do by hand, as they were in 1934, so I lean towards thinking there is a simpler way to arrive at the answer, in its simplest form. I look forward to hear your thoughts on how to solve this problem.
As basis for both solutions we use the following figure;
where $l_1$ and $l_2$ are straight lines representing the longer sides and $n_1$ and $n_2$ the perpendicular lines (called ‘normal’ in Swedish) representing the shorter sides.
Solution
Method 1
Let $l_1$ and $l_2$ have the slope $k$.
We have the following equations for the straight lines;
\begin{align*}
\text{\(l_1\):} & \quad y=k(x-2)\\
\text{\(n_1\):} & \quad y=-\tfrac{1}{k}(x-2)=-\tfrac{1}{k}x+\tfrac{2}{k}\\
\text{\(l_2\):} & \quad y=kx+\tfrac{2}{k}
\end{align*}
The $y$-coordinate of $F$ is
$$ y_F=k(x-2)\big|_{x=15}=13k $$
and the $x$-coordinate of $G$ is given by the equation
$$kx_G+\frac{2}{k}=12 \quad\Leftrightarrow\quad x_G=\frac{12-2/k}{k}.$$
The slope for $n_2$ is given by
$$ \frac{y_F-y_G}{x_F-x_G}=\frac{13k-12}{15-\tfrac{12-2/k}{k}}.$$
Since $n_1$ is parallell to $n_2$ we get
$$
\frac{13k-12}{15-\tfrac{12-2/k}{k}}=-\frac{1}{k}
\quad\Leftrightarrow\quad
13k^4-12k^3+15k^2-12k+2=0.
$$
This equation is not that easy to solve explicitly at ’first sight’, but we observe that $k=\pm i$ solves the equation and thus we can, after polynomial division, write
$$13k^4-12k^3+15k^2-12k+2=(13k^2-12k+2)(k^2+1)$$
and we find the remaining solutions to be
$$k=\tfrac{1}{13}(6\pm\sqrt{10}\,).$$
For the two $k$ we get the following figure;
The dimensions of the two rectangles are given by the ‘distance formula’ and we get
\begin{align*}
|\mathit{EF}|^2&
=(2-15)^2+(0-13k)^2\big|_{k=\frac{1}{13}(6-\sqrt{10}\,)}=215-12\sqrt{10}\\
|\mathit{EH}|^2&
=(2-0)^2+(0-\tfrac{2}{k})^2\big|_{k=\frac{1}{13}(6-\sqrt{10}\,)}=50+12\sqrt{10}\\
|\mathit{EF'}|^2&
=(2-15)^2+(0-13k)^2\big|_{k=\frac{1}{13}(6+\sqrt{10}\,)}=215+12\sqrt{10}\\
|\mathit{EH'}|^2&
=(2-0)^2+(0-\tfrac{2}{k})^2\big|_{k=\frac{1}{13}(6+\sqrt{10}\,)}=50-12\sqrt{10}
\end{align*}
The areas can now be calculated to be (in $\text{cm}^2$)
\begin{align*}
A_1&=|\mathit{EF}|\cdot|\mathit{EH}|=\sqrt{9310+1980\sqrt{10}}\\
A_2&=|\mathit{E'F'}|\cdot|\mathit{E'H'}|=\sqrt{9310-1980\sqrt{10}}
\end{align*}
This is the more ‘convoluted’ answer. It can be simplified further, but that is not obvious(?) at ‘first sight’. The simpler format is given as result in Method 2 presented below.
Method 2
We proceed as above but we calculate the areas of the 4 triangles;
\begin{align*}
A_1&=\frac{1}{2}\cdot(15-2)\cdot y_F=\frac{169}{2}k\\
A_2&=\frac{1}{2}\cdot\frac{2}{k}\cdot2=\frac{2}{k}\\
A_3&=\frac{1}{2}\cdot x_G\cdot(12-y_H)=\frac{2(1-6k)^2}{k^3}\\
A_4&=\frac{1}{2}\cdot(12-13k)\cdot\Bigl(15-\frac{12-2/k}{k}\Bigr)=\frac{(12-13k)(15k^2-12k+2)}{2k^2}
\end{align*}
and add up the total triangle area to be
$$ A=A_1+A_2+A_3+A_4=168-13k-\frac{11}{k}-\frac{12}{k^2}+\frac{2}{k^3}, $$
hence the area of the rectangle is
$$ R=12\cdot15-A=12+13k+\frac{11}{k}+\frac{12}{k^2}-\frac{2}{k^3}.$$
For the values $k=\frac{1}{13}(6\pm\sqrt{10}\,)$ we get
$$ R=90\pm11\sqrt{10}~\text{cm}^2. $$
This is the answer given in the book.
The original exam
Best Answer
The triangles $A_1,A_2,A_3$, and $A_4$ are easily seen to be similar. Let $x$ be your $y_F$. Then
$$\frac{x}{13}=\frac2{12-x}\,,$$
so $x^2-12x+26=0$, and
$$x=\frac{12\pm\sqrt{40}}2=6\pm\sqrt{10}\,.$$
The total area of the four triangles is therefore
$$13\left(6\pm\sqrt{10}\right)+2\left(6\mp\sqrt{10}\right)=90\pm11\sqrt{10}\,,$$
and the areas of the rectangles are
$$15\cdot12-\left(90\pm11\sqrt{10}\right)=90\mp11\sqrt{10}\,.$$