The limit is not defined because in order for the limit to exist, the value of the function for every possible path to $(0,0)$ must tend to the same finite value. When $y = x^2$, you have not necessarily shown that the limit is in fact $0$. When you transformed to polar coordinates and then took the limit as $r \to 0$, you are assuming that $\theta$ is a fixed constant. Therefore, you are looking only at paths that follow a straight line to the origin.
Mathematica code:
F[x_, y_] := x^2 y/(x^4 + y^2)
op = ParametricPlot3D[{r Cos[t], r Sin[t], F[r Cos[t], r Sin[t]]},
{r, 0, Sqrt[2.1]}, {t, -Pi, Pi}, PlotPoints -> 40, MaxRecursion -> 8,
Mesh -> {10, 48}, PlotRange -> {{-1, 1}, {-1, 1}, {-1/2, 1/2}},
SphericalRegion -> True, Axes -> False, Boxed -> False];
an = Show[op, ViewPoint -> {{Cos[2 Pi #], Sin[2 Pi #], 0}, {-Sin[2 Pi #],
Cos[2 Pi #], 0}, {0, 0, 1}}.{1.3, -2.4, 2}] & /@ (Range[40]/40);
Although your argument contains a grain of truth, it is not quite correct as it is written, since you wrote that the limit $\lim_{(x,y)\to (0,0)} f(x,y)$, which doesn't exist, is equal to the limit $\lim_{r \to 0} (\cdots)$, which does exist (for $\sin \theta \neq 0$) if you just view it as an ordinary single-variable limit which depends parametrically on a constant $\theta$; in fact, you just computed it yourself like that and wrote that it's equal to $1 + \cot \theta$.
The problem is that you want $\theta$ to be able to vary independently of $r$ as $r \to 0$, so you can't treat $\theta$ as a constant here. It should be viewed as an arbitrary function $\theta(r)$.
In fact, polar coordinates are mainly useful for proving that a limit exist, namely if you can write $f(x,y)$ as a bounded factor times another factor which depends only on $r$ (no $\theta$!) and which tends to zero as $r \to 0$ (really just a single-variable limit here!), then $f(x,y)\to 0$ as $(x,y)\to(0,0)$.
To show that a limit does not exist, you instead find two ways of approaching the point such that you get two different values. In your case, consider $f(t,0)$ and $f(0,t)$ as $t\to 0$, for example.
So actually I don't quite know how I would like to write the argument in a nice way if someone forced me to use polar coordinates in order to show that a limit does not exist! I would probably just write $f(x,y)$ in polar coordinates first,
$$
f(r \cos\theta(r), r \sin\theta(r)) = \cdots,
$$
(no “$\lim$” here) and then say that by making different choices of the function $\theta(r)$ (for example different constant functions!), you can make $f$ approach different values. And I would give examples of two such function $\theta(r)$ which give different limits for $f(r \cos\theta(r), r \sin\theta(r))$ as $r \to 0$.
Best Answer
If you fix $\theta$ and just let $r\to 0^+$ then you are approaching $(0,0)$ only on straight lines. This can indeed be useful in order to show that a limit does not exist, i.e. providing two different values for $\theta$ which result in two different limits. If you want to cover every path that approaches $(0,0)$ and still use polar coordinates, then you need to consider $\theta$ as a function (e.g. $\theta=\theta(r)$ arbitrary), rather than a constant. In your example, $$ \lim_{r\to 0^+}\frac{\big(r\cos\theta(r)\big)\cdot\big(r \sin\theta(r)\big)^2}{\big(r\cos\theta(r)\big)^2 + \big(r\sin\theta(r)\big)^2} = \lim_{r\to 0^+}\frac{r^3\cos\theta(r)\sin^2\theta(r)}{r^2} = \lim_{r\to 0^+}r\cos\theta(r)\sin^2\theta(r) = 0 $$
Note that considering $\theta=\theta(r)$ rather than $r=r(s)$ and $\theta=\theta(s)$ with $r(s)\to0^+$ for $s\to 0^+$ is assuming you are somehow 'strictly approaching' $(0,0)$.