Let $d(Q,DC)$ the distance between point $Q$ and the line defined by $DC$, and $d(R,DC)$ the distance between point $R$ and the line defined by $DC$.
Note that: $\triangle DQS \sim \triangle BQP$ and $\triangle BRP \sim \triangle DRT$, hence:
$$\frac{d(Q,DC)}{d(Q,AB)}= \frac{1x}{\frac{3}{2}x}= \frac{2}{3} \quad(1)$$
and
$$\frac{d(R,DC)}{d(R,AB)}= \frac{2x}{\frac{3}{2}x}= \frac{4}{3}. \quad(2)$$
From $(1)$ and $(2)$ we get:
$$d(Q,AB)=\frac{3}{5}y \quad(3)$$
and
$$d(R,AB)=\frac{3}{7}y, \quad(4)$$
which are the heights of triangles $\triangle BQP$ and $\triangle BRP$.
We can get the area of $\triangle PQR$ from:
$$A_{\triangle PQR}= A_{\triangle PQB}- A_{\triangle PRB}$$
Therefore
$$A_{\triangle PQR}= \frac{9}{70}xy=3.$$
Your teacher will hate the method in this answer, so don't use it
on the homework.
For problem 6, draw the $6,8,10$ right triangle $\triangle ACB$ as shown below:
Choosing any point $P$ along the hypotenuse $AB$ and dropping
perpendiculars from $P$ to $Q$ on $AC$ and $R$ on $BC$,
you have inscribed rectangle $PQCR$ in $\triangle ACB$ in the position desired.
Now reflect triangle $\triangle PQA$ across the line $PQ$
to obtain triangle $\triangle PQS$,
and reflect triangle $\triangle PRB$ across the line $PR$
to obtain triangle $\triangle PRT$.
Notices that if you place $P$ closer to $B$ than to $A$,
as shown in the figure, then the vertex $S$ lies outside the original
triangle $\triangle ACB$; triangles $\triangle PQS$ and $\triangle PRT$
completely cover rectangle $PQCR$ plus the triangle $\triangle SCT$,
and therefore
$$
Area(\triangle PQS) + Area(\triangle PRT) > Area(PQCR),
$$
but since
$$Area(\triangle PQS) + Area(\triangle PRT) + Area(PQCR)
= Area(\triangle ACB),$$
this implies that $$Area(PQCR) < \frac12 Area(\triangle ACB).$$
If you place $P$ closer to $A$ than to $B$, then for similar reasons
you will again find that $Area(PQCR) < \frac12 Area(\triangle ACB).$
But if you place $P$ at the midpoint of the hypotenuse $AB$,
repeating the exercise of reflecting triangles will quickly show
that $Area(PQCR) = \frac12 Area(\triangle ACB).$
This is the maximum obtainable.
For problem 7, draw the $6,8,10$ right triangle $\triangle ACB$ as shown below:
That is, drop a perpendicular from the right angle $C$ to the hypotenuse $AB$, meeting the hypotenuse at $H$. Now if we draw a rectangle as show
in the figure for problem 7, with one side lying along the hypotenuse,
part of the rectangle will lie within triangle $\triangle AHC$ and
the rest will lie within triangle $\triangle BHC$.
If the rectangle has vertices at the midpoints of legs $AC$ and $BC$,
then each part of the rectangle is half the area of the triangle within
which it lies, and therefore
$$ Area(\text{rectangle}) = \frac12 Area(\triangle ACB).$$
But if either of those vertices lies at any other point of one of the legs,
the other vertex also will lie at some point that is not the midpoint
of its leg; each part of the rectangle will have less than half
the area of the triangle in which it lies (as shown in the solution
of part 6) and therefore
$$ Area(\text{rectangle}) < \frac12 Area(\triangle ACB).$$
So once again the maximum area of the rectangle is
$\frac12 Area(\triangle ACB).$
This uses none of the standard tools of min/max problems,
just simple geometry, so it won't help you learn how to solve
min/max problems more generally.
But it may convince you that the dimensions of the rectangle for
problem 6 are $3 \times 4$
and for problem 7 are $2.5 \times 4.8$
(which is what you get when you put the vertices at the midpoints
of the applicable sides of the triangle),
so when you get these answers in the way you're supposed to,
you'll believe them, and if you get different answers then you'll know
you used the min/max methods incorrectly.
Best Answer
We have $[ABCD]=1200$, therefore the area of $\Delta{ABD}=\dfrac{1}{2}[ABCD]=600$. Now, calculate length $AD$ and $BD$. $$ \begin{align} [ABD]&=600\\ \dfrac{1}{2}AB\cdot AD&=600\\ \dfrac{1}{2}\cdot40\cdot AD&=600\\ 20\cdot AD&=600\\ AD&=30 \end{align} $$ Using Phytagoras' formula, we get $$ BD^2=AB^2+AD^2\quad\Rightarrow\quad BD=\sqrt{40^2+30^2}=50. $$ Now, calculate length $AE$. $$ \begin{align} [ABD]&=600\\ \dfrac{1}{2}BD\cdot AE&=600\\ \dfrac{1}{2}\cdot50\cdot AE&=600\\ 25\cdot AE&=600\\ AE&=24. \end{align} $$ Again we use Phytagoras' formula to obtain $BE$. $$ AB^2=AE^2+BE^2\quad\Rightarrow\quad BE=\sqrt{AB^2-BE^2}=32. $$ Thus, the area of $\Delta{ABE}$ is $$ \begin{align} [ABE]&=\dfrac{1}{2}AE\cdot BE\\ &=\dfrac{1}{2}\cdot24\cdot 32\\ &=384. \end{align} $$