By substituting $x=r\cos\theta, y=r\sin\theta$ in the formula $f(x,y,z)$, you are not converting to "polar coordinates". A polar coordinate system is a two dimensional coordinate system by definition of the term.
Then what are you doing?
Well, the conversion you made, yields a system of coordinates that is known as a cylindrical coordinate system.
Why do we convert to polar coodinates sometimes?
Because $(x,y)\to (0,0)\iff r\to 0$, assuming the canonical conversion. This can make things easier, because now we only have to consider one variable $r$ in stead of two variables $x$ and $y$. However, mind that $\lim_{r\to0}$ needs to be treated with care. See this, this and this for instance.
Did I do something wrong?
Well, not yet. The substitution you made isn't wrong, is just not necessarily useful. If you convert to cylindrical coordinates and let $r\to0$, then you are not approaching the point $(0,0)$ but the $z$-axis. So if you were to continue using this method, you would have to calculate $\lim_{(r,z)\to(0,0)}$ (also a tricky thing). Because only then are you approaching $(0,0)$.
Is there a three dimensional equivalent of the polar coordinate system?
Yes, there is. It's called the spherical coordinate system. Once you've converted from cartesian coordinates to spherical coordinates, we have that $(x,y,z)\to(0,0,0) \iff r\to 0$. Once again, it will suffice to consider only one variable $r$ now, if we're lucky (mind that $\lim_{r\to0}$ is still a tricky thing).
I did not perform any calculations on your limit. I'm leaving that as an exercise to you. I would like to leave you with the note that converting to spherical coordinate isn't a magic way to solve any $\lim_{(x,y,z)\to(0,0,0)}f(x,y,z)$ problem, The same holds for polar coordinates, or any coordinate transformation for that matter. If it works, it works. If it doesn't, too bad.
The limit is not defined because in order for the limit to exist, the value of the function for every possible path to $(0,0)$ must tend to the same finite value. When $y = x^2$, you have not necessarily shown that the limit is in fact $0$. When you transformed to polar coordinates and then took the limit as $r \to 0$, you are assuming that $\theta$ is a fixed constant. Therefore, you are looking only at paths that follow a straight line to the origin.
Mathematica code:
F[x_, y_] := x^2 y/(x^4 + y^2)
op = ParametricPlot3D[{r Cos[t], r Sin[t], F[r Cos[t], r Sin[t]]},
{r, 0, Sqrt[2.1]}, {t, -Pi, Pi}, PlotPoints -> 40, MaxRecursion -> 8,
Mesh -> {10, 48}, PlotRange -> {{-1, 1}, {-1, 1}, {-1/2, 1/2}},
SphericalRegion -> True, Axes -> False, Boxed -> False];
an = Show[op, ViewPoint -> {{Cos[2 Pi #], Sin[2 Pi #], 0}, {-Sin[2 Pi #],
Cos[2 Pi #], 0}, {0, 0, 1}}.{1.3, -2.4, 2}] & /@ (Range[40]/40);
Best Answer
If in polar coordinates the function takes the form $$ g(r) \, h(r,\theta) $$ where $g(r) \to 0$ as $r \to 0^+$ (standard single-variable limit) and the function $h$ is bounded for all $\theta$ and all $r$ in some region $0 < r < R$, then you can draw the conclusion that the two-variable limit is zero.
But that's not what you have in your case. Sure, you get a factor $r$ which tends to zero, but the remaining expression isn't bounded (as your other argument with $y=x^2$ shows; no matter how small $r$ is, you can find a $\theta$ such that the whole expression equals $1$, i.e., that other part standing together with $r$ is equal to $1/r$, which is unbounded as $r \to 0$).