If $4x+9y=60,$ then $y=\dfrac{60-4x}9$, so $xy=\dfrac{(60-4x)x}9=\dfrac{-4x^2+60x}9$
$=\dfrac{-(4x^2-60x+225)+225}{9}=\dfrac{225-(2x-15)^2}9\le\dfrac{225}9=25.$
I want to contribute my solution using geometry :D
As Cesareo told above, our problem now is to find the maximum value of $f(a,b)=a^2+b^2+a-3b$ knowing that $a^2+b^2 \leq 40$.
Geometrically, $\sqrt{a^2+b^2}$ is the distance between the point $M(a,b)$ and the origin $O(0,0)$, so the condition $a^2+b^2 \leq 40$ means that $M(a,b)$ lies inside the circle $(C)$ whose center and radius are respectively $O(0,0)$ and $\sqrt{40}$. By some calculations, we have
$$f(a,b)=\left(a+\dfrac{1}{2}\right)^2+\left(b-\dfrac{3}{2}\right)^2-\dfrac{5}{2}=MI^2-\dfrac{5}{2},$$
where $I\left(-\dfrac{1}{2},\dfrac{3}{2}\right)$. Hence, to find the maximum of $f(a,b)$ means to find the maximum of $MI$.
At this point, note that $I$ lies inside $(C)$ since $OI^2=\dfrac{5}{2}<40$. Therefore the position of $M$ maximizing $f(a,b)$ is one of two nodes of the diameter of $(C)$ which passes through $M$; that is,
$$\max f(a,b)=\max \left\{IA^2-\dfrac{5}{2},IB^2-\dfrac{5}{2}\right\},$$
where $AB$ is a diameter of $(C)$ containing $M$.
The following part is my calculations. The equation of the line $OI$ is $3x+y=0$. The coordinates of $A$ and $B$ satisfy the system
$$\begin{cases}
3x+y=0 \\
x^2+y^2=40
\end{cases} \Leftrightarrow \begin{cases}
y=-3x \\
x^2+(-3x)^2=40
\end{cases} \\ \begin{cases}
y=-3x \\
10x^2=40
\end{cases} \Leftrightarrow \begin{cases}
x=2 \\
y=-6
\end{cases} \vee \begin{cases}
x=-2 \\
y=6
\end{cases}$$
Thus $\max f(a,b)=\max \left\{IA^2-\dfrac{5}{2},IB^2-\dfrac{5}{2}\right\}=\max \left\{60,20\right\}=60.$
Hope this helps. ^^
Best Answer
We have $(a + b)(b + c)(c + a) = (a + b + c)(ab + bc + ca) - abc$ which results in $$P = \frac{1 + abc}{a + b + c}.$$
Using AM-GM, we have $$(a + b)(b + c)(c + a) \ge 2\sqrt{ab}\cdot 2\sqrt{bc}\cdot 2\sqrt{ca} = 8abc$$ and $$(a + b)(b + c)(c + a)\le \left(\frac{a + b + b + c + c + a}{3}\right)^3 = \frac{8}{27}(a + b + c)^3.$$
Thus, we have $abc \le 1/8$ and $a + b + c\ge 3/2$.
Thus, we have $$P = \frac{1 + abc}{a + b + c} \le \frac{1 + 1/8}{3/2} = 3/4.$$
Also, when $a = b = c = 1/2$, we have $(a + b)(b + c)(c + a) = 1$ and $P = 3/4$.
Thus, the maximum of $P$ is $3/4$.