For $a,b,c\geqslant 0;ab+bc+ca>0.$ Find $k_\max$ and proving in that case$:$
$$(ab+bc+ca)\Big(\dfrac{1}{(a+b)^2}+\dfrac{1}{(b+c)^2}+\dfrac{1}{(c+a)^2}\Big) \geqslant \dfrac{9}{4}+\dfrac{kabc(a^2+b^2+c^2-ab-bc-ca)}{(a+b+c)^3(ab+bc+ca)}.$$
I use computer and I found $k_\max =4.$ Then proving$:$
$$(ab+bc+ca)\Big(\dfrac{1}{(a+b)^2}+\dfrac{1}{(b+c)^2}+\dfrac{1}{(c+a)^2}\Big) \geqslant \dfrac{9}{4}+\dfrac{4abc(a^2+b^2+c^2-ab-bc-ca)}{(a+b+c)^3(ab+bc+ca)}.$$
Let $p=a+b+c,q=ab+bc+ca, r=abc,$ need to prove$:$
$$f(r)=(48q -16p^2) r^3+pq ( 23{p}^{2}-96q
) {r}^{2}+6{p}^{2}{q}^{2} ( 3{p}^{2}+8q ) r+{p
}^{3}{q}^{2} \left( {p}^{2}-4\,q \right) \left( 4{p}^{2}-q
\right) \geqslant 0.$$
Since $$f'(r)=3 \left( 48q -16\,{p}^{2}\right) {r}^{2}+2pq ( 23\,{p}^{2}
-96\,q ) r+6\,{p}^{2}{q}^{2} \left( 3\,{p}^{2}+8\,q \right)
\geqslant 0$$
$$\because (a-b)^2 (b-c)^2 (c-a)^2 \geqslant 0.$$
Therefore we need to prove $f(r)\geqslant 0$ when $r$ get the minimum values.
Let $p=1,q=\dfrac{1-t^2}{3} (0 \leqslant t \leqslant 1),r=abc$ we have$:$
$$\dfrac{1}{27} \left( 1-2t \right) \left( 1+t \right) ^{2} \leqslant r$$
(see the proof here)
So $$f(r) \geqslant f\Big(\dfrac{1}{27} \left( 1-2t \right) \left( 1+t \right) ^{2}\Big)=$$
$$={\frac {8}{19683}}\,{t}^{2} \left( 4\,{t}^{3}-6\,{t}^{2}-15\,t+49
\right) \left( t-2 \right) ^{2} \left( 2\,t-1 \right) ^{2} \left( t+
1 \right) ^{2}
\geqslant 0,$$
which is true.
But I can't prove $f'(r) \geqslant 0$ with $(a-b)^2(b-c)^2(c-a)^2\geqslant 0.$
I only check it with computer and know it's true because $\prod (a-b)^2 \geqslant 0.$
In addition$,$ I wish to know how do you find $k_\max=4$ if without computer$?$
Source. I post it to found a proof for this inequality, there was a SOS's proof.
Best Answer
nguyenhuyen_ag proved that $k\leq4$.
We'll prove that for $k=4$ our inequality is true.
Indeed, let $a+b+c=3u$, $ab+ac+bc=3v^2$, where $v\geq0$ and $abc=w^3$.
Thus, $u\geq v\geq w$ and we need to prove that: $$\frac{(ab+ac+bc)\sum\limits_{cyc}(a+b)^2(a+c)^2}{\prod\limits_{cyc}(a+b)^2}\geq\frac{9}{4}+\frac{4abc\sum\limits_{cyc}(a^2-b^2)}{(a+b+c)^3(ab+ac+bc)}$$ or $$\frac{3v^2\sum\limits_{cyc}(a^2+3v^2)^2}{(9uv^2-w^3)^2}\geq\frac{9}{4}+\frac{4w^3(9u^2-9v^2)}{81u^3v^2}$$ or $$\frac{3v^2((9u^2-6v^2)^2-2(9v^4-6uw^3)+6v^2(9u^2-6v^2)+27v^4)}{(9uv^2-w^3)^2}\geq\frac{9}{4}+\frac{4(u^2-v^2)w^3}{9u^3v^2}$$ or $f(w^3)\geq0,$ where $$f(w^3)=-16(u^2-v^2)w^9+(207u^3v^2-288uv^4)w^6+$$ $$+(1458u^4v^4+1296u^2v^6)w^3+243(3u^2-4v^2)(12u^2-v^2)u^3v^4.$$ But it's obvious that $$f'(w^3)=-48(u^2-v^2)w^6+2(207u^3v^2-288uv^4)w^3+1458u^4v^4+1296u^2v^6\geq$$ $$=-48u^2w^6-576uv^4w^3+576u^4v^4+48u^2v^6\geq0$$ (because $u\geq v\geq w$), which says that $f$ increases.
Thus, it's enough to prove our inequality for a minimal value of $w^3$,
which by $uvw$ happens in the following cases.
Let $c=0$ and $b=1$.
Thus, we need to probe that $$a\left(\frac{1}{(a+1)^2}+\frac{1}{a^2}+1\right)\geq\frac{9}{4}$$ or $$(4a^2+7a+4)(a-1)^2\geq0,$$ which is obvious.
We need to prove that: $$(2a+1)\left(\frac{2}{(a+1)^2}+\frac{1}{4}\right)\geq\frac{9}{4}+\frac{4a(a-1)^2}{(a+2)^3(2a+1)}$$ or $$(a-1)^2a^2(2a^3+13a^2+22a+12)\geq0$$ and we are done!
By the way, to find a maximal $k$, for which the inequality
$$(ab+bc+ca)\Big(\tfrac{1}{(a+b)^2}+\tfrac{1}{(b+c)^2}+\tfrac{1}{(c+a)^2}\Big) \geqslant \tfrac{9}{4}+\tfrac{ka^2b^2c^2(a-b)^2(a-c)^2(b-c)^2}{(a+b)^4(a+c)^4(b+c)^4}$$ is true for any non-negatives $a$, $b$ and $c$ such that $ab+ac+bc\neq0$,
is not so easy even with computer.