Problem. Given three non-negative numbers $a, b, c$ so that $a+ b+ c= 3,\,a^{2}+ b^{2}+ c^{2}= 5$. Prove:
$$a^{3}b+ b^{3}c+ c^{3}a\leqq 8$$
My solution in M&Y : (and I'm looking forward to seeing a nicer one(s), thanks for your interests !)
Because of the invariant of this circular permutation in $a, b, c$, suppose that $a\equiv \max\{\!a,\,b,\,c\!\}> 1$ :
$$\therefore\,ab+ bc+ ca= \frac{(a+ b+ c)^{2}- (a^{2}+ b^{2}+ c^{2})}{2}= \frac{3^{2}- 5}{2}= 2$$
Again, we have $5- a^{2}= b^{2}+ c^{2}\leqq (b+ c)^{2}= (3- a)^{2}$, therefore $a\geqq 2$ or $a\leqq 1\,(\!impossible\,\,!\!)$ .
$$\begin{align}
\therefore\,b,\,c\leqq 1\,\therefore\,a^{3}b+ b^{3}c+ c^{3}a & \leqq a^{3}b+ bc+ ca= ab(a^{2}- 1)+ 2\\
& \leqq \frac{1}{4}(a^{2}+ 4\,b^{2})(a^{2}- 1)+ 2\\
& \leqq \frac{1}{4}(5+ 3\,b^{2})(4- b^{2})+ 2\\
& = 8- \frac{1}{4}(1- b^{2})(4- 3\,b^{2})\leqq 8
\end{align}$$
The equality condition $\{\!a= 2,\,b= 1,\,c= 0\!\}\bigcup\{\!a= 1,\,b= 0,\,c= 2\!\}\bigcup\{\!a= 0,\,b= 2,\,c= 1\!\}$/q.e.d
Best Answer
Let $\{a,b,c\}=\{x,y,z\}$, where $x\geq y\geq z$.
Thus, since $x+y+z=3$ and $xy+xz+yz=2,$ we obtain: $$x+z=3-y,$$ $$xz=2-y(x+z)=2-y(3-y)=y^2-3y+2,$$ which gives $0\leq y\leq1$ because $y\geq2$ is impossible.
Thus, by Rearrangement we obtain: $$a^3b+b^3c+c^3a=a^2\cdot ab+b^2\cdot bc+c^2\cdot ca\leq x^2\cdot xy+y^2\cdot xz+z^2\cdot yz=$$ $$=y(x^3+z^3+xyz)=y\left((3-y)^3-3(y^2-3y+2)(3-y)+y(y^2-3y+2)\right)=$$ $$=y(3y^3-12y^2+8y+y).$$ Id est, it's enough to prove that $$3y^4-12y^3+8y^2+9y-8\leq0$$ or $$3y^4-3y^3-9y^3+9y^2-y^2+y+8y-8\leq0$$ or $$(y-1)(3y^3-9y^2-y+8)\leq0$$ or $$(1-y)(8(1-y)+y(3y^2-9y+7))\geq0,$$ which is true because $9^2-4\cdot3\cdot7<0.$
Done!