Hint:
Observe,
$$\begin{align*}
8(ab+bc+ca)+\sqrt{abc}&=4a(b+c)+4b(c+a)+4c(a+b)+\sqrt{abc} \\
(b+c-a)^2&=(1-2a)^2=1-4a(1-a)=1-4a(b+c)
\end{align*}$$
Using the Cauchy-Schwarz inequality,
$$\sum_{\text{cyc}}\sqrt{\left((b+c-a)^{2}+4a(b+c)\right)\left(abc+4a(b+c)\right)}\geq \sum_{\text{cyc}} \left(\sqrt{abc}(b+c-a)+4a(b+c)\right)\\
\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;=8(bc+ca+ab)+\sqrt{abc}.$$
Marginally less ugly, let $s=a+b, p=ab$, then:
$$
a^4+b^4 = 17 \;\;\iff\;\; (s^2-2p)^2-2p^2=17 \;\;\iff s^4 - 4ps^2 + 2p^2 - 17 = 0
$$
Solving the quadratic in $\,p\,$, and retaining the root which satisfies $p \le s^2$:
$$
p = s^2 - \sqrt{\frac{s^4+17}{2}}
$$
The inequality to prove is equivalent to:
$$
(15s-17)^2 \ge 2\cdot 14^2 \,p = 14^2\left(2s^2-\sqrt{2\left(s^4+17\right)}\right)
$$
Rearranging with positive quantities on both sides and squaring:
$$
2 \cdot 14^4 \left(s^4+17\right) \ge \left(2 \cdot 14^2 s^2 - (15s-17)^2\right)^2
$$
After expanding, collecting and "luckily" finding the rational root $s=3$:
$$
17 (2879 s^4 - 10020 s^3 - 9622 s^2 + 17340 s + 71919) \ge 0
\\ \iff\;\;\;\; (s - 3)^2 (2879 s^2 + 7254 s + 7991) \ge 0
$$
The quadratic factor has no real roots, so the inequality holds true, with equality iff $s=3\,$, which then gives $p=2$ i.e. $\{a,b\}=\{1,2\}$.
Best Answer
Due to @Calvin Lin, I will post solution based on his hint later.
Now I get solution by AM-GM. Notice that: $\frac{1}{a^2}+a^2=\left(\frac{1}{a}+a\right)^2-2=\left(\frac{1}{a}+a+\sqrt{2}\right)\left(\frac{1}{a}+a-\sqrt{2}\right)$
Then using AM-GM: $$2\frac{\sqrt{2}\frac{1}{a^2}}{\sqrt{\frac{1}{a^2}+a^2}}+\frac{\frac{1}{a}+a+\sqrt{2}}{2+\sqrt{2}}+\frac{\frac{1}{a}+a-\sqrt{2}}{2-\sqrt{2}}\ge\frac{4}{a}$$ Or:$$\frac{\sqrt{2}\frac{1}{a^2}}{\sqrt{\frac{1}{a^2}+a^2}}+a\ge\frac{1}{a}+1$$ Similarly, $$\frac{\sqrt{2}b^2}{\sqrt{\frac{1}{a^2}+a^2}}+a+\frac{1}{a}\ge2b+1$$ The rest is obvious: $$\frac{\frac{1}{a^2}+b^2}{\sqrt{\frac{1}{a^2}+a^2}}\ge\sqrt{2}(b-a+1)$$