Yes, you're right, but if you want to be precise you should keep track of how big the remainder terms are when making these approximations. Let's call the numerator $f(x,y)$. Then, as $(x,y)\to (0,0)$,
\begin{align}
f(x,y)&=\left[x+ O(x^3)\right]^{\alpha}\left[\frac{y^2}{2}+O(y^4)\right]^{\beta}\\
&=\frac{x^{\alpha}y^{2\beta}}{2^{\beta}}\underbrace{[1+O(x^2)]^{\alpha}[1+O(y^2)]^{\beta}}_{:=\rho(x,y)}
\end{align}
Therefore,
\begin{align}
\frac{f(x,y)}{(x^2+y^2)^{4}}&=\frac{x^{\alpha}y^{2\beta}}{2^{\beta}(x^2+y^2)^4}\cdot \rho(x,y),
\end{align}
whereby from the definition of $\rho$, it is evident that $\lim\limits_{(x,y)\to(0,0)}\rho(x,y)=1$. Now, as you mention,
- if $\alpha+2\beta>8$, the first factor approaches $0$; so the fact that $\rho(x,y)\to 1$ (in particular it is bounded near the origin) implies by the squeeze theorem that the product also approaches $0$.
- If $\alpha+2\beta=8$, then the limit does not exist because the limit along the line $x=y$ is non-zero... it is infact equal to $\frac{1}{2^{\beta+4}}$. On the other hand, $\alpha+2\beta=8>0$ and $\alpha,\beta\geq 0$ imply that one of these quantities is strictly positive. If $\alpha>0$, then take the limit along the line $x=0$, while if $\beta>0$ take the limit along $y=0$. In any case, we have two paths, along one we have a non-zero limit, along the other we have a zero limit.
- If $\alpha+2\beta<8$, then the absolute value of the product approaches $\infty$.
Just in case you're not familiar with the big O notation, when I write $O(x^2)$ above, what I mean is that it is equal to a function $\phi(x)$, such that there exist $M>0,\delta>0$ such that for all $|x|\leq \delta$, we have $|\phi(x)|\leq M|x^2|$.
Note that nowhere in my answer did I use any $\sim$ symbols. They are all truly equal signs (once we understand the meaning of big O), and each equal sign captures exactly "how large" the remaining terms are. So, this really is completely formal. If this is your first time seeing this, I would suggest reading section 3.5, on infinitesimals, of Loomis and Sternberg's Advanced Calculus (atleast up to theorem 5.1).
Since they already talked about your procedure in the comments, I shall talk about some nasty tricks I still remember. I love how "shall" sounds in English, so be prepared!
Polar coordinates
Now I could say "The power in the numerator is higher than the denominator which means the numerator goes faster to 0, and the limit exists". But in order to prove it mathematically, I tried to do this:
A spot-on observation on your part, as it should be verified each time; powers tend to behave weirdly in multivariate limits, especially when at the denumerator and non-balanced, like a $x^4+y^2$. In my experience one of the most useful instruments is the conversion to polar coordinates - see this discussion for example. As you may know the issue with multivariable calculus is that you now have more than the usual two directions of approach, and the conversion in polar coordinates tends to be very useful. Look at this limit:
$$ \lim_{(x,y)\to(0,0)} \frac{xy^2}{x^2+y^2}$$
if you try polar coordinates $x=r\cos\theta$ and $y=r\sin\theta$ you should get something like
$$ \lim_{r\to0^+} \frac{r\cos\theta(\sin\theta)^2}{1}$$
where I write $r\to0^+$ because it is implied by $(x,y)\to(0,0)$. The limit of course tends to zero because whatever (accepted) value $\theta$ has gets destroyed by $r$ on the long - or short? - run. You could be even more specific using the squeeze theorem since of course
$$-1\leq\sin\theta,\cos\theta\leq+1$$
for our $\theta\in[0,2\pi]$.
The preliminary substitution
Note that you have multiple directions to approach your point from, so you can play dirty with your limit; I think you already know this, but I shall mention it for whoever may stumble here with the same question. My advice is to use this as soon as you read the exercise.
Suppose your point is, like in your exercise, $P=(0,0)$. Then you can try substituting into the limit the values $(x,y)$ of any series approaching to $(0,0)$, like $(t^2,t)$ for $t\to 0$. This also renders the limit an usual one-variable calculation. The point of this all is that if you try different "directions" (i.e. a different series approaching to $P$) and get different results your limit does not exist.
Approach via family of lines
I have also seen a similiar method using a family of lines $y-y_0=m(x-x_0)+q$ passing by your $P=(x_0,y_0)$, but I find it a tad cumbersome. It is useful when your limit is asking for a change of coordinates, like
$$\lim_{(x,y)\to(2,1)}\frac{(x-2)^2(y-1)}{(x-2^4+(y-1)^2}$$
Here you shall use
$$y=m(x-2)+1$$
where $m$ is a real parameter, then
$$\lim_{(x,y)\to(2,1)}\frac{(x-2)^2(y-1)}{(x-2)^4+(y-1)^2}=\lim_{x\to2}\frac{m(x-1)}{m^2+(x-2)^2}=0$$
since it does not depend on the value of $m$. This of course does not mean this limit exits, you are only approaching it with lines, there are infinitely many other ways to approach it. In fact you could note this limit doesn't exist using different substitutions. Hint: you already tried a linear approach, try something... less linear!
This method is a tad boring and time consuming, so I much prefer the substitution $x'=x-x_0$, $y'=y-y_0$ but it is worth mentioning since sometimes it can come in handy. The substitution could even make the expression of your limit much worse, and while trying to evaluate the differentiability of a function via limits it's not something that you really want.
I just want to note that polar coordinates tend to confuse while getting the hand of them... look at the linked discussion for example. You must be careful your limit does not depend from the direction you approach from, so be aware of what $\theta$ stands for in $(r,\theta)$ coordinates. It's usual to be fooled by expressions like
$$\lim_{r \to 0} \frac{r (\cos^2\theta\sin\theta)}{r^2\cos^4\theta + \sin^2\theta}$$
The approved answer and the related comments explain it better than I ever could.
Best Answer
The examples you give do not involve homogeneous polynomials. Your professor's tip does not apply.
Meanwhile $$ (x^4 \pm y^9)^2 \geq 0 $$ $$ x^8 \pm 2x^4 y^9 + y^{18} \geq 0 $$ $$ x^8 + y^{18} \geq 2 x^4 |y^9 | $$ $$ \frac{2 x^4 |y|^9}{x^8 + y^{18}} \leq 1 $$ $$ \frac{2 x^4 |y|^{13}}{x^8 + y^{18}} \leq y^4 $$
I went through Lagrange multipliers for $$ f(x,y) = \frac{x^a y^b}{x^{2c} + y^{2d}} $$ with $a,b,c,d $ positive integers.
When $ad-2cd+bc=0$ the ratio $f$ is bounded but has no limit.
When $ad-2cd+bc<0$ there is no limit and the ratio is not bounded.
When $ad-2cd+bc>0$ the limit is $0$
Your question has $a=4, b=13, c=4, d=9$ so that $ad-2cd+bc=36- 72 +52 =16 > 0 $ so the limit is zero.