Although your argument contains a grain of truth, it is not quite correct as it is written, since you wrote that the limit $\lim_{(x,y)\to (0,0)} f(x,y)$, which doesn't exist, is equal to the limit $\lim_{r \to 0} (\cdots)$, which does exist (for $\sin \theta \neq 0$) if you just view it as an ordinary single-variable limit which depends parametrically on a constant $\theta$; in fact, you just computed it yourself like that and wrote that it's equal to $1 + \cot \theta$.
The problem is that you want $\theta$ to be able to vary independently of $r$ as $r \to 0$, so you can't treat $\theta$ as a constant here. It should be viewed as an arbitrary function $\theta(r)$.
In fact, polar coordinates are mainly useful for proving that a limit exist, namely if you can write $f(x,y)$ as a bounded factor times another factor which depends only on $r$ (no $\theta$!) and which tends to zero as $r \to 0$ (really just a single-variable limit here!), then $f(x,y)\to 0$ as $(x,y)\to(0,0)$.
To show that a limit does not exist, you instead find two ways of approaching the point such that you get two different values. In your case, consider $f(t,0)$ and $f(0,t)$ as $t\to 0$, for example.
So actually I don't quite know how I would like to write the argument in a nice way if someone forced me to use polar coordinates in order to show that a limit does not exist! I would probably just write $f(x,y)$ in polar coordinates first,
$$
f(r \cos\theta(r), r \sin\theta(r)) = \cdots,
$$
(no “$\lim$” here) and then say that by making different choices of the function $\theta(r)$ (for example different constant functions!), you can make $f$ approach different values. And I would give examples of two such function $\theta(r)$ which give different limits for $f(r \cos\theta(r), r \sin\theta(r))$ as $r \to 0$.
The inequality you have written is wrong. Cosine is not an incerasing function so $x\leq y$ does not imply $\cos\, x \leq \cos \, y$. Instead, use the inequality $1-\cos \theta \leq \frac {\theta^{2}} 2$ valid for all real $\theta$. Proof of this inequality: $1-\cos \theta -\frac {\theta^{2}} 2$ vanishes at $0$ and its derivative is $\sin\, \theta - \theta$ which is negative for all $\theta >0$. Hence the inequality holds for all positive $\theta$. Since both sides are even functions the inequality holds for all $\theta$.
Best Answer
The proof with polar is messy, but I will finish it, and then provide an alternative approach.
Polar Approach:
The key is to consider the cases when $\sin^2(\theta)<\rho$ and $\sin^2(\theta)\ge\rho$.
When $\sin^2(\theta)<\rho$ we have
$$\frac{\rho^2|\cos^3(\theta)\sin(\theta)|}{\rho^2\cos^4(\theta)+\sin^2(\theta)}<\frac{\rho^3|\cos^3(\theta)|}{\rho^2\cos^4(\theta)}=\frac\rho{\sqrt{1-\sin^2(\theta)}}<\frac\rho{\sqrt{1-\rho}}\to0$$
When $\sin^2(\theta)\ge\rho$ we have
$$\frac{\rho^2|\cos^3(\theta)\sin(\theta)|}{\rho^2\cos^4(\theta)+\sin^2(\theta)}<\frac{\rho^2}{\sin^2(\theta)}\le\frac{\rho^2}\rho=\rho\to0$$
Hence the limit is $0$, independent of $\theta$.
Alternative Approach:
Polar relies on $\rho^2=x^2+y^2$, which, in this case, results in a fairly messy expression. Alternatively, we can let $\rho^2=x^4+y^2$ to match our denominator. We thus have the inequalities $\rho^2\ge x^4$ and $\rho^2\ge y^2$, or $|x|\le\sqrt\rho$ and $|y|\le\rho$ (since $\rho>0$). Then we have
$$\frac{|x^3y|}{x^4+y^2}\le\frac{(\sqrt\rho)^3\rho}{\rho^2}=\sqrt\rho\to0$$
which is a much cleaner proof.