Let $k>1$ and $b=c=a$.
Thus, $d=4-3a$, $0\leq a\leq\frac{4}{3}$ and we need $$3a^2+(4-3a)^2+4(\sqrt3-1)(a^3(4-3a))^k\geq2\sqrt{3(a^3+3a^2(4-3a))}$$ or
$$3a^2-6a+4-a\sqrt{3(3-2a)}+(\sqrt3-1)(a^3(4-3a))^k\geq0$$ or
$$\frac{(4-3a)(4-9a+6a^2-3a^3)}{3a^2-6a+4+a\sqrt{3(3-2a)}}+(\sqrt3-1)(a^3(4-3a))^k\geq0$$ or
$$\frac{4-9a+6a^2-3a^3}{3a^2-6a+4+a\sqrt{3(3-2a)}}+(\sqrt3-1)a^{3k}(4-3a)^{k-1}\geq0,$$ which is wrong for $a=\frac{4}{3},$ which says that $k\leq1$.
We'll prove that $k=1$ is valid.
Indeed, let $a=\min\{a,b,c,d\}$ and $$f(a,b,c,d)=a^2+b^2+c^2+d^2+4(\sqrt3-1)abcd-2\sqrt{3(abc+abd+acd+bcd)}.$$
Thus, $0\leq a\leq1$ and
$$f(a,b,c,d)-f\left(a,\frac{b+c+d}{3},\frac{b+c+d}{3},\frac{b+c+d}{3}\right)=$$
$$=b^2+c^2+d^2-\frac{(b+c+d)^2}{3}-4(\sqrt3-1)a\left(\left(\frac{b+c+d}{3}\right)^3-bcd\right)+$$
$$+2\sqrt3\left(\sqrt{\frac{a(b+c+d)^2}{3}+\frac{(b+c+d)^3}{27}}-\sqrt{abc+abd+acd+bcd}\right)=$$
$$=\frac{2}{3}\sum_{cyc}(b^2-bc)-4(\sqrt3-1)a\left(\left(\frac{b+c+d}{3}\right)^3-bcd\right)+$$
$$+\frac{\frac{2a}{\sqrt3}\sum\limits_{cyc}(b^2-bc)+2\sqrt3\left(\left(\frac{b+c+d}{3}\right)^3-bcd\right)}{\sqrt{\frac{a(b+c+d)^2}{3}+\frac{(b+c+d)^3}{27}}+\sqrt{abc+abd+acd+bcd}}\geq$$
$$\geq\frac{2}{3}\sum_{cyc}(b^2-bc)-4(\sqrt3-1)a\left(\left(\frac{b+c+d}{3}\right)^3-bcd\right)+$$
$$+\frac{\frac{2a}{\sqrt3}\sum\limits_{cyc}(b^2-bc)+2\sqrt3\left(\left(\frac{b+c+d}{3}\right)^3-bcd\right)}{4}=$$
$$=\left(\frac{2}{3}+\frac{a}{2\sqrt3}\right)\sum_{cyc}(b^2-bc)+\left(\frac{\sqrt3}{2}-4(\sqrt3-1)a\right)\left(\left(\frac{b+c+d}{3}\right)^3-bcd\right).$$
Now, if $\frac{\sqrt3}{2}-4(\sqrt3-1)a\geq0$ so $f(a,b,c,d)-f\left(a,\frac{b+c+d}{3},\frac{b+c+d}{3},\frac{b+c+d}{3}\right)\geq0.$
But for $\frac{\sqrt3}{2}-4(\sqrt3-1)a\leq0$ since by Schur $$\left(\frac{b+c+d}{3}\right)^3-bcd\leq\frac{4}{27}\sum_{cyc}(b^3-bcd),$$
we obtain:
$$f(a,b,c,d)-f\left(a,\frac{b+c+d}{3},\frac{b+c+d}{3},\frac{b+c+d}{3}\right)\geq$$
$$\geq\left(\frac{2}{3}+\frac{a}{2\sqrt3}\right)\sum_{cyc}(b^2-bc)+\left(\frac{\sqrt3}{2}-4(\sqrt3-1)a\right)\cdot\frac{4}{27}\sum_{cyc}(b^3-bcd)=$$
$$=\left(\frac{2}{3}+\frac{a}{2\sqrt3}+\left(\frac{\sqrt3}{2}-4(\sqrt3-1)a\right)\cdot\frac{4}{27}(4-a)\right)\sum_{cyc}(b^2-bc)\geq0$$ for any $0\leq a\leq1.$
Id est, $$f(a,b,c,d)\geq f\left(a,\frac{b+c+d}{3},\frac{b+c+d}{3},\frac{b+c+d}{3}\right),$$ which says that it's enough to prove our inequality for equality case of three variables.
Let $b=c=a$.
Thus, $d=4-3a$, where $0\leq a\leq \frac{4}{3}$ and we need to prove that:
$$3a^2+(4-3a)^2+4(\sqrt3-1)a^3(4-3a)\geq2\sqrt{3(a^3+3a^2(4-3a))},$$ which after squaring of the both sides gives:
$$(a-1)^2(4-3a)(4-a+2(4\sqrt3-5)a^3-4(2-\sqrt3)a^4-6(2-\sqrt3)a^5)\geq0,$$ which is true for $0\leq a\leq \frac{4}{3}$.
Best Answer
Another way which works for several such inequalities, consider minimising $RHS - LHS$ (note there will exist a minimum for this function in this domain). At the point of minimum, if possible let there be two unequal variables, WLOG say $a \neq b$. Then consider $$RHS - LHS = 16\left(ab(c+d)+cd(a+b)\right)+5(a+b+c+d) - 16\left(ab+(a+b)(c+d)+cd \right) \\= -16(1-c-d)\color{red}{ab} + (\color{red}{a+b})\left(cd+5-16(c+d)\right)+5(c+d)-16cd$$
Now it can be seen that replacing both $a, b$ with $\frac12(a+b)$ will change only the first term, and in fact it will reduce $RHS-LHS$ as $ab$ will increase. Hence at the minimum, we must have $a=b=c=d$, so it is enough to check the inequality for this case, which is a lot easier.
$16(6a^2) \leqslant 5(4a) + 16(4a^3) \iff 4a(1-4a)(5-4a) \geqslant 0$ which is true for $a \in [0, \frac14]$.