Let $a=b=c=\frac{2}{3}$. Thus, $P=8.$
We'll prove that it's a minimal value of $P$.
Indeed, we need to prove that
$$\sum_{cyc}(4a^3+5abc)\geq(a+b+c)^3$$ or
$$3\sum_{cyc}(a^3-a^2b-a^2c+abc)\geq0,$$ which is true by Schur.
Let $k$ be a maximal value.
Thus, $k$ it's a minimal number, for which the inequality
$$\left|\sum_{cyc}(a^3b-a^3c)\right|\leq k(a^2+b^2+c^2)^2$$ is true for all reals $a$, $b$ and $c$. We see that $k>0$.
Now, let $a+b+c=3u$, $ab+ac+bc=3v^2$, where $v^2$ can be negative, $abc=w^3$ and $u^2=tv^2$.
Thus, $$|(a+b+c)(a-b)(a-c)(b-c)|\leq k(a^2+b^2+c^2)^2$$ or
$$(a+b+c)^2\prod_{cyc}(a-b)^2\leq k^2(a^2+b^2+c^2)^4$$ or
$$9u^2\cdot27(3u^2v^4-4v^6-4u^3w^3+6uv^2w^3-w^6)\leq81k^2(3u^2-2v^2)^4$$ or
$$u^2w^6+2u^3(2u^2-3v^2)w^3-3u^4v^4+4u^2v^6+\frac{k^2}{3}(3u^2-2v^2)^4\geq0,$$ for which we need
$$u^6(2u^2-3v^2)^2-u^2\left(-3u^4v^4+4u^2v^4+\frac{k^2}{3}(3u^2-2v^2)^4\right)\leq0$$ or
$$t^2(2t-3)^2+3t^2-4t\leq\frac{k^2}{3}(3t-2)^4$$ or
$$\frac{12t(t-1)^3}{(3t-2)^4}\leq k^2.$$
Consider two cases.
- $t>0.$
Thus, $t\geq1$ and in this case
$$\frac{12t(t-1)^3}{(3t-2)^4}\leq\lim_{t\rightarrow+\infty}\frac{12t(t-1)^3}{(3t-2)^4}=\frac{4}{27}.$$
Indeed, $$\frac{4}{27}-\frac{12t(t-1)^3}{(3t-2)^4}=\frac{4(27t^3-27t^2-15t+16)}{27(3t-2)^4}=\frac{4((t-1)(27t^2-15)+1)}{27(3t-2)^4}>0.$$
- $t<0$.
Thus, by AM-GM
$$\frac{12t(t-1)^3}{(3t-2)^4}=\frac{-3t(2-2t)^3}{2(3t-2)^4}\leq\frac{\left(\frac{-3t+3(2-2t)}{4}\right)^4}{2(3t-2)^4}=\frac{81}{512}$$ and since
$$\frac{81}{512}>\frac{4}{27},$$ we obtain:$$k^2=\frac{81}{512}$$ and $$k=\frac{9}{16\sqrt2}.$$
The equality occurs for $$-3t=2-2t$$ or $$t=-2,$$ which is possible, which says that $\frac{9}{16\sqrt2}$ is the answer.
Best Answer
Also, smoothing helps.
Indeed, let $k$ be a minimal value of the expression $\frac{3a^2+b^2+3c^2}{ab+ac+bc}$ and let $a+c=constant.$
Thus,$k>0$ and we see that it's enough to prove the inequality $$3(a+c)^2-6ac+b^2\geq k(ac+b(a+c))$$ for a maximal value of $ac$, which happens for $a=c$.
Thus, we need to find a maximal value of $k$ for which the inequality $$6a^2+b^2\geq k(b^2+2ab)$$ or $$(6-k)a^2+b^2\geq2kab$$ is true for any positives $a$ and $b$.
Since should be $6-k>0,$ by AM-GM we obtain: $$(6-k)a^2+b^2\geq2\sqrt{6-k}ab,$$ which gives that should be $$2\sqrt{6-k}ab\geq2kab,$$ or $$k^2+k-6\leq0$$ or $$k\leq2.$$
The equality occurs for $a=c$ and $b=2c$, which says that $2$ is a minimal value and $a^2\cdot2a=432,$ which gives $(a,b,c)=(6,12,6)$ and $$3a+b+c=18+12+6=36.$$