The most important thing is practice. As you calculate more and more limits, you'll start to develop some form of intuition regarding what method to try first, and the correct approach will probably come faster to you. You might see a problem and be reminded of some other limit you did before, so you'll try a similar method and maybe find it also works.
As you've said, appealing directly to the definition is sometimes the most straightforward way. Some inequalities which are often useful include $$|x|≤\sqrt{x^2+y^2} \ ; \ |\sin (u)|≤|u| \ ; \ 2|xy|≤x^2 +y^2 \ ; \ |x+y|≤|x|+|y| $$
A very common way to show that a given limit does NOT exist is to approach to the limit point trough several paths which conduct to different values. For instance, consider: $$\lim_{(x,y)\to (0,0)}{3y^2-2x^2 \over 7x^2+3y^2}$$
Putting $y=x$ we obtain: $$\lim_{x \to 0}{3x^2 -2x^2 \over 7x^2+3x^2}=\lim_{x \to 0}{x^2 \over 10x^2}={1 \over 10}$$
On the other hand, if we get closer to $(0,0)$ by moving along the horizontal axis, that is $y=0$ we get: $$\lim_{x \to 0}{-2x^2 \over 7x^2}={-2 \over 7}$$
It follows that the limit does not exist. What we have done is actually a shorthand for finding two sequences $p_n$ and $q_n$ such that $p_n \to (0,0)$ and $q_n \to (0,0)$ but $$\lim_{n \to \infty}f(p_n)\neq \lim_{n \to \infty}f(q_n)$$ This is usually a quicker method than applying the definition, but it only works for showing that the limit does not exist. Even if you were to try a thousand paths which all led to the same value, that would not show that the limit actually exists. If might, however, suggest you it does, and so it might be time to try and prove it using the definition.
Some commonly used paths include: $x=0$, $y=0$, $x=y$, $y=x^n$, $y=mx$, where $m\in \mathbb{R}$.
A third method that is also quite common is changing to polar coordinates. In the example you gave, put: $$\begin{cases} x=r\cos \theta \\ y=r\sin \theta
\end{cases}$$ Then $\frac{xy(x^2-y^2)}{x^2+y^2}$ becomes $${r^4\cos\theta \sin \theta(\cos^2 \theta - \sin ^2 \theta) \over r^2(cos^2 \theta + \sin^2 \theta)}=r^2\cos\theta \sin \theta(\cos^2 \theta - \sin ^2 \theta)$$
This tends to $0$ as $r \to 0^+$, hence the desired limit equals $0$. What we have used is that the vector $(x,y)$ tends to $0$ if and only if the radius gets arbitrarily small, that is $r→0^+$ (it tends to zero from the right because the radius is always positive by definition). To see this, you can note that: $\left \|(x,y)\right \|=\sqrt{x^2+y^2}=\sqrt{r^2(\cos^2\theta +\sin^2\theta)}=r$.
Other, more obscure, techniques include using the mean value theorem or a Taylor expansion (be it one or several variables) to deduce some useful inequality or identity. Its probably better not to worry about these and try to solve them when you encounter them, as they are not very common.
Best Answer
Note that $f(\sqrt{2}) = 0$, and that $$ x^2-2 = (x-\sqrt{2})(x+\sqrt{2}), $$ so that $$ |-x^2+2 - 0| = |x^2-2 - 0| = |x-\sqrt{2}||x+\sqrt{2}|. $$ Note that when $x$ is close to $\sqrt{2}$, the right factor is close to $\sqrt{2}$, and the left factor is small.
Can you find your delta using this?