Let $B'$ be the foot of the perpendicular from $B$ onto $AC$
When right triangle $\triangle ABC$ is divided into two triangles by $BB'$, they are similar. In this case, we also know that they have inscribed circles with radius $r=3$ and with radius $R=4$, so they are similar with a $3:4$ ratio. Therefore $AB : BC = 3 : 4$.
Therefore $AB : BC : AC = AB' : B'B : AB = BB' : B'C : BC = 3 : 4 : 5$.
In a triangle with sides $3,4,5$ the inradius is $1$. So here $\triangle AB'B$ has sides $9,12,15$ and $\triangle BB'C$ has sides $12,16,20$. We can figure out many side lengths from here.
Let $T', H', S'$ be the feet of the perpendiculars from $T, H, S$ onto $AB$. We know $\triangle AT'T$ is a $3:4:5$ right triangle, and $AT = AK = AB' - KB' = 9-3 =6$, so $TT' = 4.8$ and $AT' = 3.6$; also, $T'K = AK - AT' = 6-3.6 = 2.4$. Since $TT' : T'K = 2 : 1$ and $\triangle HH'K \sim \triangle TT'K$, we know $HH' : H'K = 2:1$ as well.
Let's do the same thing from the other side, looking at $\triangle CSS'$, $\triangle QSS'$, and $\triangle QHH'$. We get $SS' = 7.2$ and $QS'=2.4$, so $SS' : QS' = 3:1$ and therefore $HH' : QH' = 3:1$ as well.
It follows that $H'K : HH' : QH' = 3 : 6 : 2$ so $H'K : QH' = 3 : 2$. But we know $H'K + QH' = 7$, so $H'K = 4.2$ and $QH' = 2.8$, and therefore $HH' = 8.4$. The area of the triangle we want is $\frac12 (7)(8.4) = 29.4$.
Here is a geometrical solution without much algebra.
For a right triangle, if you draw a line through the points of tangency of the incircle with the perpendicular sides, it does bisect the arcs of the circumcircle on both sides. In other words, $M$ and $N$ are midpoints of minor arcs $AB$ and $BC$ respectively. At the end of the answer, I have shown a proof.
With that, note that $\triangle BTM \sim \triangle NKB$. That leads to,
$\frac{r}{a} = \frac{b}{r} \implies r = \sqrt{ab}$
As $FM$ is perpendicular bisector of $AB$ and $FN$ is perpendicular bisector of $BC$,
$\frac{AB}{2} = \sqrt{ab} + \frac{a}{\sqrt2}$ and $\frac{BC}{2} = \sqrt{ab} + \frac{b}{\sqrt2}$
As we know $AB$ and $BC$ in terms of $a$ and $b$, we are done, for $S_{\triangle ABC} = \frac 12 \cdot AB \cdot BC$.
Proof of the property that I used in the above answer -
Say $M$ and $N$ are midpoints of the arcs $AB$ and $BC$ and segment $MN$ intersects $AB$ and $BC$ at $T$ and $K$ respectively.
$\angle BIN = 45^\circ + \angle A/2$
So, $\angle KPN = 90^\circ + \angle A/2$
Also, $\angle PNK = \angle C/2$
That leads to $\angle BKM = \angle PKN = 45^\circ$
Also note that $\angle INK = \angle ICK = \angle C / 2$ so $ICNK$ is cyclic and therefore $\angle KIN = \angle KCN = \angle A / 2$
That leads to $\angle IKM = \angle A / 2 + \angle C / 2 = 45^\circ$
$\angle BKI = \angle BKM + \angle IKM = 90^\circ$.
So $K$ must be point of tangency of incircle with side $BC$. Finally since $KI \parallel BT$ and $BT = BK = KI$, $T$ is the point of tangency of incircle with side $AB$.
Best Answer
While you are starting with a formula for the pentagon in terms of $R$, you can derive directly as follows -
Given $a$ is diagonal,
$ \displaystyle S_{AED} = S_{BCD} = \frac 12 \cdot a \cdot \frac {a \tan36^0}{2}$
$ \displaystyle S_{ADB} = \frac 12 \cdot a^2 \sin36^\circ$
So, $~ \displaystyle S = \frac{a^2}{2} \sin 36^\circ \cdot \frac{1 + \cos36^\circ}{\cos36^\circ}$
As $\cos 36^\circ = \dfrac{1 + \sqrt5}{4}$
$ \displaystyle \sin 36^\circ = \sqrt{1 - \frac{6+2 \sqrt5}{16}} = \frac 12 \cdot \sqrt{\frac{5-\sqrt5}{2}}$
Also, $\dfrac{1 + \cos36^\circ}{\cos36^\circ} = \dfrac{5 + \sqrt5}{1 + \sqrt5} = \sqrt5$
So, $ \displaystyle S = \frac{a^2}{4} \sqrt{\frac{25-5\sqrt5}{2}}$