Overview
In full generality, proving things like this can be very hard. But
when you have a nice function and a nice region of $\mathbb{R}^{n}$,
there's a standard toolbox that I think works well in practice. Basically:
- Split up the region if necessary to reduce the dimension to make it
more convenient to work with.
- Then compactify the region by adding "points at infinity" in an appropriate way (which way depends a bit on the context) or using a trick with arctan to achieve the same result.
- If the function can be extended to a continuous function on the compact region, then an extremum must exist, and you can use Multivariable Calculus techniques (e.g. look for critical points, parametrize the boundary and/or use Lagrange multipliers) to find candidates. Then just compare values of all of the candidates.
- If the function can't be so extended, or it's too much effort, you may still be able to show that there is a bound on the function over a part of the region, and then cut the bad part(s) out of consideration so that you reduce to the above.
There are many different ways this might look like in practice,
so I'll present various ways that come to mind for tackling the problem
at hand, as examples. In actuality, almost all of this is overkill
for the specific original problem.
Horizontal Strips
For this particular function, the domain is two-dimensional, but it
might be easier to manage if we cut it into one-dimensional horizontal
strips of the form $\left[-t,t\right]\times\left\{ t\right\} $ for
$t\ge0$.
Initial Work
Note that each strip is compact, so there is a maximum on each strip.
If we can find the maxima and compare them, we can solve both parts.
On a horizontal strip, the function reduces to $g_{t}(x)=4x^{2}+9t-\frac{1}{3}t^{3}$.
Since we know this is a parabola, we can simply note that the maximum
is attained at $x=\pm t$ and the value is $g_{t}(t)=4t^{2}+9t-\frac{1}{3}t^{3}$.
(In general, we could differentiate with respect to $x$ to obtain
$g_{t}'(x)=8x$ so that the points worth checking are the critical
point $x=0$ and the endpoints $x=\pm t$.)
Now that we have the maximum on each strip, we must see if and where
there is a height $y=t_{0}$ (where $t\in\left[0,\infty\right)$)
where this maximum $h(t)=4t^{2}+9t-\frac{1}{3}t^{3}$ is maximized.
Unfortunately, $\left[0,\infty\right)$ is not compact, so we can
either prove a bound to cut off some of it, or compactify it.
Prove a Bound
Without using any geometric properties of cubics, we could examine
the derivative $h'(t)=-t^{2}+8t+9$. The only positive zero is at
$t=9$, after which it's negative (since it's continuous and $h'(10)=-11<0$).
Therefore, $h(t)$ is decreasing for $t\ge9$. In general, if the
behavior of the derivative $h'(t)$ were more complicated, we could
then restrict our attention to the compact $[0,9]$. But since we
can see that $h'(t)$ is positive on $[0,9]$, we immediately know
that a maximum is attained at $t=9$.
Putting everything together, the maximum is $h(9)=\boxed{162\text{ at }(\pm9,9)}$.
Compactification
Note that ${\displaystyle \lim_{t\to\infty}}h(t)=-\infty$ since $-\frac{1}{3}t^{3}$
dominates for large $t$. And $h(0)=0$. If we add in $\pm\infty$
from the extended reals,
then we can work properly with the compact interval $[0,\infty]$.
The only other candidate for a maximum would be where $h'(t)=0$,
at $t=9$. Since $h(9)=162$ is greater than the endpoint values of
$0$ and $-\infty$, it must be the maximum.
Alternatively, since $\tan$ is increasing on $\left(-\pi/2,\pi/2\right)$,
we can define $\widetilde{h}(t)=\arctan\left(4\tan^{2}t+9\tan t-\frac{1}{3}\tan^{3}t\right)$
on $t\in\left[0,\pi/2\right)$ which has a maximum at $t_{0}$ exactly
if $h(t)$ has a maximum at $\tan t_{0}$. But since ${\displaystyle \lim_{x\to\pi/2^{-}}}\tan x=\infty$
and ${\displaystyle \lim_{x\to-\infty}}\arctan x=-\pi/2$, $\widetilde{h}$
can be extended to a continuous function $[0,\pi/2]\to\mathbb{R}$,
no $\infty$ required. Then the values of $\widetilde{h}$ at the
endpoints are $0$ and $-\pi/2$, both of which are less than the
value of $\arctan(162)$ at the zero of $\widetilde{h}'$, $t=\arctan9$.
Either way, again we have that the maximum of the original functions
is $\boxed{162\text{ at }(\pm9,9)}$.
Vertical Strips
Another approach would be to cut the region into 1-dimensional vertical
strips of the form $\left\{ t\right\} \times\left[\left|t\right|,\infty\right)$
for some real $t$. On each strip, we the function reduces to $g_{t}(y)=4t^{2}+9y-\frac{1}{3}y^{3}$
for $y\ge|t|$.
Compactification
One of these strips can be compactified in much the same way as discussed
earlier: either being comfortable with the extended reals, or replacing
$g_{t}(y)$ with $\widetilde{g_{t}}(y)=\arctan\left(4t^{2}+9\tan y-\frac{1}{3}\tan^{3}y\right)$
for $y\ge\arctan\left|t\right|$. We have potential extrema at the
endpoints, and where $g_{t}'=0$. Note that ${\displaystyle \lim_{y\to\infty}}g_{t}(y)=-\infty$
so that we have continuity (though there won't be a maximum there),
and $g_{t}\left(|t|\right)=4t^{2}+9|t|-\frac{1}{3}|t|^{3}$. $g_{t}'(y)=9-y^{2}$
has a $0$ at $y=3$ (where $g_{t}(y)=4t^{2}+18$) if $|t|\le3$ and
no zeros if $|t|>3$. Note that $4t^{2}+18\ge4t^{2}+|t|\left(9-\frac{1}{3}t^{2}\right)$
for $\left|t\right|\le3$, so the maximum of $g_{t}(y)$ is attained
at $y=\begin{cases}
3 & \text{ if }|t|\le3\\
|t| & \text{ if }|t|\ge3
\end{cases}$. The values attained are $h(t)=\begin{cases}
4t^{2}+18 & \text{ if }|t|\le3\\
4t^{2}+|t|\left(9-\frac{1}{3}t^{2}\right) & \text{ if }|t|\ge3
\end{cases}$.
Now that we have the maximum on each strip, we must see if and where
there is a horizontal position $x=t_{0}$ where this maximum $h(t)$
is maximized. Since $4t^{2}+18$ is increasing as $|t|$ increases,
it achieves its maximum on $\left[-3,3\right]$ at $t=\pm3$. Therefore,
it suffices to consider $h(t)=4t^{2}+|t|\left(9-\frac{1}{3}t^{2}\right)$
on. Since $h$ is an even function, we really just have to worry about $h(t)=4t^{2}+9t-\frac{1}{3}t^{3}$
on $\left[3,\infty\right)$. Above, we found that this function's
maximum on $\left[0,\infty\right)$ is attained at $9$. So using
a similar approach (perhaps looking at $h(3)$ instead of $h(0)$),
we would reach the same conclusion, and the maximum of the original
function is still $\boxed{162\text{ at }(\pm9,9)}$.
Prove a Bound
Instead of compactifying the whole vertical strip, we could have noticed
that $g_{t}'(y)<0$ for $y>3$, so that $g_{t}(y)\le g_{t}\left(\max\left(3,\left|t\right|\right)\right)$
and we only need to consider the compact part of the strip where $y\in\left[|t|,\max\left(3,\left|t\right|\right)\right]$.
Maximizing $g_{t}$ on this interval and then maximizing the maxima
is very similar to the above, now.
Diagonal Strips
Given the shape of the domain, one more approach to reduce the dimension
would be to cut it into 1-dimensional diagonal strips of the form
$\left\{ \left(s\cos t,s\sin t\right):s\in\left[0,\infty\right)\right\} $
for $t\in\left[\pi/4,3\pi/4\right]$. On each strip, we the function
reduces to $g_{t}(s)=4\left(s\cos t\right)^{2}+9\left(s\sin t\right)-\frac{1}{3}\left(s\sin t\right)^{3}$
for $s\ge0$.
As before, we could compactify/notice that $g_{t}(s)$ is eventually
very negative since $\sin t>0$ if $t\in\left[\pi/4,3\pi/4\right]$.
So to find candidates for a maximum, we only need to examine $s=0$,
(where $g_{t}(s)=0$) and where $g_{t}'(s)=0$. $g_{t}'(s)=-s^{2}\sin^{3}t+8s\cos^{2}t+9\sin t$.
The only positive zero is at $s_{t}=\dfrac{-8\cos^{2}t-\sqrt{64\cos^{4}t+36\sin^{4}t}}{-2\sin^{3}t}=\dfrac{4\cos^{2}t+\sqrt{16\cos^{4}t+9\sin^{4}t}}{\sin^{3}t}.$
Setting $r=\sqrt{16\cos^{4}t+9\sin^{4}t}$ for convenience, we find
that $g_{t}(s_{t})$ is the following nonnegative quantity (and hence
this is the maximum):
$$h(t)=4\left(\dfrac{4\cos^{2}t+r}{\sin^{3}t}\cos t\right)^{2}+9\left(\dfrac{4\cos^{2}t+r}{\sin^{3}t}\sin t\right)-\frac{1}{3}\left(\dfrac{4\cos^{2}t+r}{\sin^{3}t}\sin t\right)^{3}$$
$$
=\cdots
$$
$$
=\dfrac{2}{3\sin^{2}t}\left(4\cos^{2}t+r\right)\left(9+8\dfrac{\cos^{4}t}{\sin^{4}t}+2\dfrac{\cos^{2}t}{\sin^{4}t}r\right)
$$
Now, the domain of $t$ is the compact interval $\left[\pi/4,3\pi/4\right]$,
so we can optimize $h(t)$ relatively easily. At the endpoints where
$t=\pi/2\pm\pi/4$, we have $h(t)=162$ and $s_{t}=9\sqrt{2}$, corresponding
to the points $\left(\pm9,9\right)$ in the plane. We also have $h'(t)$
equal to a very complicated expression. But $h''(t)$ is a littler
simpler can can be seen to be positive on $\left[\pi/4,3\pi/4\right]$,
so that $h'(t)$ has at most one zero. In fact, $h'(\pi/2)=0$, and
we have $h(\pi/2)=18<162$. Once again, the maximum is $\boxed{162\text{ at }(\pm9,9)}$.
The Whole Region#
All of the previous methods focused on slicing up the two-dimensional
region. But we can deal with both dimensions at once.
Prove a Bound
Note that if $|x|\le y$, then $x^{2}\le y^{2}$, so that $f(x,y)=4x^{2}+9y-\frac{1}{3}y^{3}\le4y^{2}+9y-\frac{1}{3}y^{3}$.
Since ${\displaystyle \lim_{y\to\infty}}4y^{2}+9y-\frac{1}{3}y^{3}=-\infty$,
it suffices to look at a compact region of the form, say, $\left|x\right|\le y\le6+3\sqrt{7}$
since $6+3\sqrt{7}$ is the largest zero of $4y^{2}+9y-\frac{1}{3}y^{3}$
(beyond which it must be negative), and we know that $f(0,1)=26/3>0$.
Therefore, $f$ has an absolute maximum somewhere in that large triangle
of height $6+3\sqrt{7}$, either at a critical point, or along the
boundary. As noted in the question, $\left(0,3\right)$ is the only
critical point in the interior of the triangle. We have $f\left(0,3\right)=18>0$,
so no point on the top edge (where $f\left(x,y\right)\le0$) is relevant.
It remains to examine the bottom edges for candidates. Since $f$
is symmetric about $x=0$, it suffices to check the right edge $\left\{ (x,x):x\in\left[0,6+3\sqrt{7}\right]\right\} $.
This can be done either by examining $f(x,x)$ directly, or by using
Lagrange multipliers to handle $f(x,y)$ with the constraint $y=x$.
(With a more complicated curve like a circle $x^{2}+y^{2}=9$, aside
from Lagrange Multipliers, you could parametrize with a new parameter
as in $f\left(3\cos t,3\sin t\right)$, or break it up into separate
functions of $x$, as in $f\left(x,\sqrt{9-x^{2}}\right)$ and $f\left(x,-\sqrt{9-x^{2}}\right)$.)
Either way, the boundary point $(0,0)$ of that boundary edge is a
candidate for a maximum (the other boundary point was ruled out already).
Looking at $f(x,x)$ directly is similar to work we've done above
and involves $\frac{\mathrm{d}}{\mathrm{d}x}f(x,x)=0$. Lagrange Multipliers
would have us examine the gradient of $y-x$, and solve $8x=\lambda\left(-1\right)$,
$9-y^{2}=\lambda\left(1\right)$, and $y-x=0$, which yields the same
equation as $\frac{\mathrm{d}}{\mathrm{d}x}f(x,x)=0$. In any case,
we find candidates $(\pm9,9)$.
Taking all the candidates together, we find that the maximum must
be one of $f\left(0,3\right)=18$, $f(0,0)=0$, and $f(\pm9,9)=162$.
Since $162>18>0$, we have the maximum is $\boxed{162\text{ at }(\pm9,9)}$.
Compactify
Because a bound was so easy to find, it's not worth compactifying
in two dimensions, but I include this for completeness and a nice
graphical perspective.
Because of the extra dimension, the easiest way to make sure we are
adding on acceptable "points at infinity" is to use the arctan
trick. Define $\widetilde{f}\left(x,y\right)=\arctan\left(4\tan^{2}x+9\tan y-\frac{1}{3}\tan^{3}y\right)$
for $|x|\le y<\pi/2$, so that $\widetilde{f}$ has a global maximum
at $\left(x,y\right)$ if $f$ does at $\left(\tan x,\tan y\right)$.
We can extend the domain of $\widetilde{f}$ to include the top of
the triangle at $y=\pi/2$, via $\widetilde{f}\left(x,y\right)=-\pi/2$
if $y=\pi/2$. It turns out that $\widetilde{f}$ is then continuous
on the entire compact triangle, for instance via the multivariable
squeeze theorem and
$$
-\pi/2\le\widetilde{f}(x,y)\le\arctan\left(4\tan^{2}y+9\tan y-\frac{1}{3}\tan^{3}y\right)\underset{y\to\pi/2^{-}}{\to}-\pi/2
$$
Now we can do the same sort of argument as before, noting that at
the critical point we have $\widetilde{f}(0,\arctan3)=\arctan(18)>-\pi/2$,
so the top edge is not the source of a candidate for the maximum.
We can switch back to $f$ or stay with $\widetilde{f}$ when finding
candidates, and arrive at the same result as every other time for
the original function: $\boxed{162\text{ at }(\pm9,9)}$.
Graphs
One benefit of $\widetilde{f}$ is that it essentially allows us to
graph all of $f$ at once.
Here are two views of $\widetilde{f}$.
To understand how steep that cliff is, here is $\widetilde{f}(t,t)=\arctan\left(4\tan^{2}t+9\tan t-\frac{1}{3}\tan^{3}t\right)$
zoomed in near its zero of $t\approx1.49917$
Best Answer
Note that the equation is a circle with center $O(5,0)$ and radius $3$: $$x^2 -10x+y^2 +16=0 \iff (x-5)^2+y^2=9$$ The objective function is $\frac yx=k \iff y=kx$, whose contour lines will pass through the origin. So you need to find the slope of the tangent to the circle. See the graph:
$\hspace{4cm}$
Hence, the slope is $k=\frac 34$, which is the maximum value of $\frac yx$ at $x=\frac{16}{5}$ and $y=\frac{12}{5}$.