If the line has to pass through $(0,b)$, $(3,2)$ and $(a,0)$ it has to satisfy the following equation:
$$
y-0=\dfrac{2-0}{3-a}(x-a)
$$
Observe that when $x=0$, $y=-\dfrac{2}{3-a}a$, that is $b=-\dfrac{2}{3-a}a$
Now if both $a,b$ are non-negative, then certainly the line will have negative slope (why?). Now the area $A(a,b)$ of the triangle formed by the $x$-axis, the $y$-axis and the line, is given by $A(a,b)=\dfrac{1}{2}ab$. By replacing the value of $b$ in this equation we obtain a formula for the area of the triangle as a function on $a$, $A(a)=-\dfrac{1}{2}a\dfrac{2}{3-a}a=\dfrac{a^2}{a-3}$. Now if you want to know for what value of $a$ this function has a maximum or a minimum, differential calculus is of great help.
$$
A'(a)=\dfrac{2a(a-3)-a^2}{(a-3)^2}=-\dfrac{a^2-6a}{(a-3)^2}
$$
To find the critical points of this function we have to find the zeroes of $A'(a)$, that is, we need to find the solutions of the following equation:
$$
-\dfrac{a^2-6a}{(a-3)^2}=0
$$
In this case the zeroes will come from the zeroes of the numerator that is $a^2-6a=0$, but those are $a=0,\quad a=6$, We have to reject $a=0$ since it does not produces a triangle. The answer is $a=6$. So the value of the area will be $A(6)=\dfrac{1}{2}\dfrac{6^2}{(6-3)^2}=2$. In order to find out whether this is a maximum or a minimum you could use the second derivative criteria:
$$
A''(a)=-\dfrac{4(a-3)^3(a^2-6a)-(a-3)^4(2a-6)}{(a-3)^4}=-\dfrac{2(a-3)^3(2a^2-12a-(a-3)(a-3)}{(a-3)^4}=-\dfrac{2a^2-12a-a^2+6a-9}{(a-3)^3}=-\dfrac{a^2-6a-9}{(a-3)^3}
$$
So $A''(6)=-\dfrac{6^2-6(6)-9}{(6-3)^3}=\dfrac{9}{27}=\dfrac{1}{3}>0$, the area is a minimum.
By similar triangles, it can be proven that for $(x_0,y_0=f(x_0))$ with corresponding vertices $(x_c,0)$, and $(0,y_c)$ all lying on the tangent line, for any $y=f(x_0)$,
$$\frac{x_0}{x_c}+\frac{y_0}{y_c}=1$$
$x_c$, $y_c$, and $y_0$ can all be expressed as functions of $x_0$, so can their ratios.
The sum of the ratios in the above equation add up to one. As one ratio increases, the other decreases.
Principles of Linear Programming suggest finding optimal values where component terms take on extreme values or component terms are equal.
$x_c$ or $y_c$ take on extreme values when the opposite coordinate nears a vertex. Under these conditions, the area under the associated triangle gets arbitrarily large.
So the extremes will not give a minimum.
Now suppose $\frac{x_0}{x_c}=\frac{y_0}{y_c}$ from the other principle of Linear Programming.
Then $\frac{x_0}{x_c}=\frac{y_0}{y_c}=\frac{1}{2}$.
So $x_c=2x_0$ and $y_c=2y_0$, making $(x_0,y_0)$ the midpoint of the hypotenuse. By Thales' Theorem, the midpoint of a hypotenuse is equidistant from all vertices of the right triangle.
The area of the corresponding triangle is $\frac{1}{2}x_cy_c=\frac{1}{2}\cdot4x_0y_0=2x_0y_0$
So the triangle's area is minimized when the area enclosed by the rectangle with opposite corners the origin and the point on $f(x_0)$ is a minimum.
$$A(x)=x(1-x^2)=x-x^3$$
Which has a minimum at $1-3x^2=0$
We also know the length of the hypotenuse is twice the length of the distance of $(x_0,y_0)$ to the origin at the point of optimal area.
This means the intercepts must lie on a circle centered at $(x_0,y_0)$ and passing through the origin. On intercept will always be outside of the circle, and one always in. This allows for a searching algorithm with straight edge and compass.
Best Answer
the line passing through $(1,8)$ has equation
$y-8=m(x-1)\;\;y=mx-m+8$
which intersect $x-$axis at $\left(\dfrac{m-8}{m};\;0\right)$ and $y-$axis at $(0;\;8-m)$
Hypotenuse $h(m)=\sqrt{\left(\dfrac{m-8}{m}\right)^2+(8-m)^2}$
as square root is an increasing function, $h(m)$ will be minimum when
$r(m)=\left(\dfrac{m-8}{m}\right)^2+(8-m)^2=m^2+\dfrac{64}{m^2}-16 m-\dfrac{16}{m}+65$
will be minimum
$r'(m)=-\dfrac{128}{m^3}+\dfrac{16}{m^2}+2 m-16=\dfrac{2 \left(m^4-8 m^3+8 m-64\right)}{m^3}$
$m^4-8 m^3+8 m-64=0\to (m-8) (m^3-8) =0$
$m=8$ gives hypotenuse $h(8)=0$ which makes no sense
$m^3=8\to m=2$ gives $h(2)=3 \sqrt{5}$ which is the minimum we were looking for
indeed second derivative is $r''(m)=\dfrac{2 \left(m^4-16 m+192\right)}{m^4}$ and $r''(2)=22>0$
it is positive at $m=2$ so it is a minimum
hope this helps