This is probably overkill, but if the polygon is convex and you fix the axis along which the rectangle is to be aligned, the problem is a second order cone program. If $x$ is a 2-d vector, any convex polygon can be expressed as the set of points satisfying $Ax \le b$, where A is a matrix and b is a vector. (Why? each bounding line is an inequality $a_i^T x \le b_i$, so then let $a_i^T$ be the ith row of $A$, and $b_i$ the ith component of b.)
So if $x$ is a corner of your rectangle, and $\ell,w$ are the length and width of your rectangle, the problem is:
$$
\begin{eqnarray}
\max_{x,\ell, w} \ \ell w \\
Ax \le b, \quad
A\left(x + \left[\begin{matrix} 0 \\ w\end{matrix}\right] \right) \le b,
\quad
A\left(x + \left[\begin{matrix} \ell \\ 0\end{matrix}\right] \right)& \le b,
\quad
A\left(x + \left[\begin{matrix} \ell \\ w\end{matrix}\right] \right)& \le b,
\quad \ell,w \ge 0
\end{eqnarray}
$$
Then there's a standard trick to turn a hyperbolic constraint into an SOCP:
$$\left \|
\left[\begin{matrix} 2z \\ \ell - w\end{matrix}\right] \right\|_2 \le \ell + w \Leftrightarrow z^2 \le \ell w
$$
So instead of maximizing $\ell w$ you can include the above constraint, and maximize $z$. If you don't have access to an SOCP solver is probably more complicated than what you need, but it does show how one can solve the problem exactly.
Writing $a := |BC|$, $b := |CA|$, $c := |AB| = \sqrt{a^2+b^2}$, and $r = |PE| = |PF|$ (so that $|PD| = 3r$), we have
$$\begin{align}
|\triangle ABC| &= |\triangle ABP| + |\triangle BCP| + |\triangle CAP| \\[4pt]
\implies \qquad \frac{1}{2} |BC||CA| &= \frac{1}{2} \left(\; |AB| |PF| + |BC||PD| + |CA||PE| \;\right) \\[4pt]
\implies \qquad a b &= c r + 3 a r + b r = r ( 3 a + b + c )\\[6pt]
\implies \qquad r &= \frac{ab}{3 a + b + c} = \frac{ab}{3 a + b + \sqrt{a^2+b^2}}
\end{align}$$
To address @DanielV's suggestion of generalizing to higher dimensions, consider a right-corner tetrahedron $OABC$, with right corner at $O$ and edge lengths $a := |OA|$, $b := |OB|$, $c := |OC|$. (Note that I'm changing notation slightly from the above.) Let a sphere with center $P$ and radius $r$ be tangent to the faces around vertex $A$, and let a congruent sphere (tangent to the first) be tangent to the faces around vertex $O$. Then $P$ has distance $r$ from faces $\triangle OAB$, $\triangle OCA$, $\triangle ABC$ (the ones touching $A$), and distance $3r$ from face $\triangle OBC$ (the one opposite $A$).
Here's a poor attempt at a diagram:
(In this case, the altitudes from $P$ are color-coded to match their parallel counterparts through $O$. The black altitude is to face $\triangle ABC$.)
Thus,
$$\begin{align}
|OABC| &= |OABP| + |OBCP| + |OCAP| + |ABCP| \\[4pt]
\implies \qquad \frac{1}{6}a b c &= \frac{1}{3}\left(\; r\;|\triangle OAB| + r \;|\triangle OCA| + r\;|\triangle ABC| + 3r\;|\triangle OBC| \;\right) \\[4pt]
&= \frac{1}{3}r \cdot \frac{1}{2} \left(\; a b + c a + 3 b c + 2\;|\triangle ABC| \;\right) \\[6pt]
\implies \qquad r &= \frac{abc}{3bc + ab + ca + 2\;|\triangle ABC|} \qquad (\star)
\end{align}$$
Fun fact: The Pythagorean Theorem for Right-Corner Tetrahedra says that
$$|\triangle ABC|^2 = |\triangle OBC|^2 + |\triangle OCA|^2 + |\triangle OAB|^2$$
so that we have
$$|\triangle ABC| = \frac{1}{2} \sqrt{\; b^2 c^2 + c^2 a^2 + a^2 b^2 \;}$$
and $(\star)$ becomes
$$r = \frac{abc}{3bc + ab + ca + \sqrt{\; b^2 c^2 + c^2 a^2 + a^2 b^2 \;}}$$
In $4$-dimensional space (where there's an analogous Pythagorean Theorem, as there is in any-dimensional space), we have
$$r = \frac{abcd}{3bcd + acd + abd + abc + \sqrt{\;b^2 c^2 d^2 + a^2 c^2 d^2 + a^2 b^2 d^2 + a^2 b^2 c^2\;}}$$
and so forth.
Incidentally, the matching-notation version of the initial answer is
$$r = \frac{ab}{3b + a + \sqrt{\;b^2 + a^2\;}}$$
Best Answer
You know what a picture frame looks like, right? Two concentric rectangles.
You are given the dimensions of the outer rectangle. You are given the area of the inner rectangle.
You are asked to find the thickness of the border between them.
$$20\textrm{cm}\left\{ \vphantom{\bbox[green, 2ex, border:solid 1pt]{\bbox[white, 1ex, border:solid 1pt]{\begin{array}{l}\qquad\\160\textrm{cm}^2\\~\\~\end{array}}}} \right. % \overbrace{\bbox[green, 2ex, border:solid 1pt]{\bbox[white, 1ex, border:solid 1pt]{\begin{array}{l}\qquad\\160\textrm{cm}^2\\~\\~\end{array}}}}^{14\textrm{cm}}$$