Let $t=x+y$ and we need $t_{\min}$. Then we have $$(t+y)(t-y)y=2\implies t^2y-y^3=2$$ and thus $$t^2 = y^2+{2\over y}$$ so if we apply Am-Gm for three terms we get $$t^2=y^2+{1\over y}+{1\over y}\geq 3$$
and minimum value $t=\sqrt{3}$ is achieved iff $y^2 = {1\over y}$ i.e. $y=1$ and $x=\sqrt{3}-1$.
To use inequalities to find minima, we need the equality condition to be satisfied as well, hence in this case we need $x^2=8x=\frac{32}x$, which doesn't have a solution.
If you have a "lucky guess" for which $x$ the minimum occurs (and you know you have the minimum value somehow), we can split the terms to equal components and then apply AM-GM. If not, there is still a way - consider the more general version:
$$x^2 + \alpha \; \frac{8x}\alpha + \beta\; \frac{64}{\beta x^3} \geqslant (1+\alpha+\beta) \left(x^2\cdot \left(\frac{8x}\alpha\right)^\alpha \cdot \left( \frac{64}{\beta x^3}\right)^\beta\right)^{\frac1{1+\alpha+\beta}} \tag{1}$$
While it is not necessary to write the above inequality out, I am reproducing it for clarity.
Now for using this inequality for minimum, the RHS must be a constant, i.e. $x-$free, so we need $\alpha = 3\beta-2$ considering the exponents of $x$.
Further, for the equality condition we must have some $x$ for which
$$x^2 = \frac{8x}\alpha = \frac{64}{\beta x^3} \iff x = \frac{8}{3\beta-2} = \frac{64(3\beta-2)^4}{\beta \cdot 8^4}$$
The last equality gives $(3\beta-2)^5=2^9\beta$ which can be solved (well, by observation, or trying one's luck at rational roots) for $\beta=2$. After doing all that, you can write out ($1$) with the correct $\alpha, \beta$ values for the desired AM-GM.
Best Answer
Hint: If we additionally make the (reasonable, though not necessarily true) assumption that the minimum happens when $ a = b$, then reduce it to a 1 variable inequality and hence show that the minimum is 4 obtained at $ ( 1/3, 1/3, 4/3)$.
Note that we've not proven this is the minimum yet.
Use this to figure out what AM-GM's to create.
Modifying your approach, we have the following inequalities:
$a^2 + b^2 \geq 2ab \longrightarrow$ This reinforces $a=b$ at the equality case.
$Xa^2 + 0.5c^2 \geq 2\sqrt{X/2}ac \longrightarrow$ What does the (assumed) equality case tell us about $X$?
$Yb^2 + 0.5c^2 \geq 2\sqrt{Y/2}bc \longrightarrow$ What does the (assumed) equality case tell us about $Y$?
From the equality case, we require
Then, weight the inequalities to get
$$ 10a^2 + 10b^2 + c^2 \geq 4(ab+bc+ca) = 4,$$
with equality when $ a = b = c/4, ab+bc+ca = 1$, IE $ ( 1/3, 1/3, 4/3)$.
(Thankfully, our initial assumption is true.)
I wanted to show you how to derive these inequalities (assuming that your stated approach could work). It isn't just "magic" or "by observation" or "by luck".
For the actual solution, you just need to write out the 3 AM-GM inequalities, sum them up, and verify the equality case.
Using this approach, try your hand at minimizing (say) $3a^2 + 2b^2 + c^2 $ given $ab+bc+ ca = 1$.
(Note: I've not actually done this, so I can't guarantee that the values will look nice.)