Define
$$
f(a,\lambda) = -\frac{a}{a^{11}+1} + \lambda \log(a) + \frac{1}{2}
$$
Then, for any choice of $\lambda$,
$$
f(a,\lambda) + f(b,\lambda) + f(c,\lambda) = -\frac{a}{a^{11}+1} -\frac{b}{b^{11}+1} -\frac{c}{c^{11}+1} + \frac{3}{2}
$$
and we need to show that this is $\ge 0$.
It suffices to show that, for some $\lambda^*$ and for all $a$, $f(a, \lambda^*) \ge 0$.
Clearly, for any lambda, $f(a=1,\lambda) = 0$. In order to keep $f(a,\lambda) $ positive for $a >1$ and $a <1$, we demand
$$
0 = \frac{d f(a,\lambda)}{d a}|_{a=1}
$$
which results in $\lambda^* = - \frac94$. We therefore investigate
$$
f(a,\lambda^*) = -\frac{a}{a^{11}+1} -\frac{9}{4} \log(a) + \frac{1}{2}
$$
By inspection, we have that $f(a,\lambda^*) \ge 0$ for $a\in (0, 1.1]$. So the inequality is obeyed at least for $a,b,c < 1.1$, and it remains to be shown that the inequality is obeyed outside this specification.
This gives rise to three cases:
case 1: $a,b,c > 1.1$. This is not possible since $abc = 1$.
case 2: $a < 1.1$ ; $b,c > 1.1$. Now observe two facts:
By inspection, $ \frac{a}{a^{11}+1} < 0.75$ for any $a$.
For $b > 1.1$, $ \frac{b}{b^{11}+1} \le \frac{1.1}{1.1^{11}+1} \simeq 0.2855$ since $ \frac{b}{b^{11}+1}$ is falling for $b > 1.1$.
Hence, in case 2, $ \frac{a}{a^{11}+1} + \frac{b}{b^{11}+1}+ \frac{c}{c^{11}+1} < 0.75 + 2\cdot 0.2855 = 1.3210 < \frac32$ which proves case 2.
case 3: $a,b < 1.1$ ; $c > 1.1$. Here $abc = 1$ requires $a\cdot b =1/c < 1.1^{-1} = 0.909$. Also note that, for some given $c$, $1/(1.1 c) <a<1.1$ in order to observe $a,b < 1.1$. Following case 2, we have that $f(c) = \frac{c}{c^{11}+1} $ is falling with $c$. These conditions could be further exploited (this has not yet been pursued in the comments).
As Martin R. poined out, the maximum will be attained at a point where at least two out of $a,b,c$ equal. In this case, this would be $a=b$. So we can consider proving
$$
g(a) = \frac32 - \frac{2 a}{a^{11}+1} - \frac{a^{-2}}{a^{-22}+1} \ge 0
$$
for $a < 1/\sqrt{1.1} \simeq 0.9535$.
Note that in this range, the minimum of $g(a)$ occurs at $a^*\simeq 0.8385$ and has a value of $g(a^*) \simeq 0.00525$. Other than this inspection of the function $g(a)$, I couldn't offer a better proof.
Let $a+b=c=3u$, $ab+ac+bc=3v^2$ and $abc=w^3$.
Thus, we need to prove that $f(w^3)\geq0,$ where
$$f(w^3)=\frac{u^3v^2}{w^3}-\left(\frac{1}{2}+\frac{5\sqrt3}{18}\right)(3u^2-2v^2).$$
But we see that $f$ decreases, which says that it's enough to prove our inequality for a maximal value of $w^3$, which happens for equality case of two variables.
Let $b=a$ and $c=3-2a$, where $0<a<\frac{3}{2}.$
Id est, it's enough to prove that
$$\frac{2}{a}+\frac{1}{3-2a}\geq\left(\frac{1}{2}+\frac{5\sqrt3}{18}\right)(2a^2+(3-2a)^2),$$ which is smooth.
I got that the last inequality is equivalent to
$$((10+6\sqrt3)a^2-(23+13\sqrt3)a+12+8\sqrt3)(2a-3+\sqrt3)^2\geq0,$$ which is obvious.
Best Answer
By AM-GM $$\sum_{cyc}\sqrt{\frac{a+b}{b+1}}\geq3\sqrt[6]{\prod\limits_{cyc}\frac{a+b}{a+1}}.$$ Thus, it's enough to prove that $$(a+b)(a+c)(b+c)(3abc+1)^2\geq16a^2b^2c^2(a+1)(b+1)(c+1).$$ Now, let $a+b+c=3u$, $ab+ac+bc=3v^2$ and $abc=w^3$.
Thus, we need to prove that $$(9uv^2-w^3)(3w^3+1)^2\geq16w^6(w^3+3v^2+3u+1)$$ and since by AM-GM $uv^2\geq w^3,$ it's enough to prove that $$uv^2(3w^3+1)^2\geq2w^6(w^3+3v^2+3u+1),$$ which is true by AM-GM.
Indeed, by AM-GM $$2w^6(w^3+3v^2+3u+1)=2w^9+6v^2w^6+6uw^6+2w^6\leq$$ $$\leq2uv^2w^6+6uv^2w^5+6uv^2w^4+2uv^2w^3$$ and it's enough to prove that: $$(3w^3+1)^2\geq2w^6+6w^5+6w^4+2w^3$$ or $$(3w^3+1)^2\geq2w^3(w+1)^3.$$ Can you end it now?