You don't need to change to polar coordinates. Notice that at any point $(x,y)\ne (0,0)$ the function is continuous since it is a quotient of continuous functions. So, you just need to check the limit as $(x,y)$ goes to $(0,0)$. Noting that as $(x,y)$ goes to $(0,0)$ so does $x^2+y^2$, and since $\lim_{t\to 0}\sin(t)/t=1$ you may conclude that the limit of the function as $(x,y)\to (0,0)$ is equal to $0$ (note the multiplication by $x$ in the numerator). Since $f(0,0)=0$ be definition it follows that the function is continuous everywhere.
By substituting $x=r\cos\theta, y=r\sin\theta$ in the formula $f(x,y,z)$, you are not converting to "polar coordinates". A polar coordinate system is a two dimensional coordinate system by definition of the term.
Then what are you doing?
Well, the conversion you made, yields a system of coordinates that is known as a cylindrical coordinate system.
Why do we convert to polar coodinates sometimes?
Because $(x,y)\to (0,0)\iff r\to 0$, assuming the canonical conversion. This can make things easier, because now we only have to consider one variable $r$ in stead of two variables $x$ and $y$. However, mind that $\lim_{r\to0}$ needs to be treated with care. See this, this and this for instance.
Did I do something wrong?
Well, not yet. The substitution you made isn't wrong, is just not necessarily useful. If you convert to cylindrical coordinates and let $r\to0$, then you are not approaching the point $(0,0)$ but the $z$-axis. So if you were to continue using this method, you would have to calculate $\lim_{(r,z)\to(0,0)}$ (also a tricky thing). Because only then are you approaching $(0,0)$.
Is there a three dimensional equivalent of the polar coordinate system?
Yes, there is. It's called the spherical coordinate system. Once you've converted from cartesian coordinates to spherical coordinates, we have that $(x,y,z)\to(0,0,0) \iff r\to 0$. Once again, it will suffice to consider only one variable $r$ now, if we're lucky (mind that $\lim_{r\to0}$ is still a tricky thing).
I did not perform any calculations on your limit. I'm leaving that as an exercise to you. I would like to leave you with the note that converting to spherical coordinate isn't a magic way to solve any $\lim_{(x,y,z)\to(0,0,0)}f(x,y,z)$ problem, The same holds for polar coordinates, or any coordinate transformation for that matter. If it works, it works. If it doesn't, too bad.
Best Answer
Note: this answer stems from the (now removed) comments left under my answer to your other question about polar coordinates. As requested, I have edited to add the proofs.
Before proceeding, let us define (possibly non-standard notation) $$\lim_{(r,\phi)\to\{0\}\times\mathbb R}g(r,\phi)=L\tag{$\diamond$}$$ to mean that for each $\epsilon>0$, there is an open neighborhood $U$ of $\{0\}\times\mathbb R$ in $[0,\infty)\times\mathbb R$ such that $(r,\phi)\in U$ and $r\neq0$ implies $|g(r,\phi)-L|<\epsilon.$
In the case $g(r,\phi)=f(r\cos\phi,r\sin\phi)$, which is $2\pi$-periodic in the second argument, this can be simplified, by requiring instead that for each $\epsilon>0$, there is a $\delta>0$ such that $0<r<\delta$ implies $|g(r,\phi)-L|<\epsilon$. (Intuitively, in this case we do not have to worry about $\delta$ getting smaller and smaller as $\phi\to\pm\infty$.) This can be done e.g. by using the tube lemma from elementary topology.
We may now state:
Proposition. The following are equivalent for a function $f:\mathbb R^2\setminus\{(0,0)\}\to\mathbb R$:
Proof. The equivalence of 1 and 2 is standard and follows directly from the definitions.
To see that 2 implies 5, let $p:[0,\infty)\times\mathbb R\to\mathbb R^2$ be defined by $p(r,\phi)=(r\cos\phi,r\sin\phi).$ This is obviously continuous, so if $\tilde f$ is continous, $g=\tilde f\circ p$ must be continuous as well.
Next, we show that 5 implies 3. To see this, let $\epsilon>0$ and $\phi_0\in\mathbb R$. By $5$, there is a $\delta>0$ such that $\|(r,\phi)-(0,\phi_0)\|<\delta$ implies $|g(r,\phi)-g(0,\phi_0)|<\epsilon$. In particular, if $r>0$, this means that $\|(r,\phi)-(0,\phi_0)\|<\delta$ implies $|f(r\cos\phi,r\sin\phi)-L|<\epsilon$, which is precisely 3.
Next, we show that 3 implies 4. Let $\epsilon>0$. For each $\phi_0\in\mathbb R$ we have a $\delta(\phi_0)>0$ such that $\|(r,\phi)-(0,\phi_0)\|<\delta(\phi_0)$ implies $|f(r\cos\phi,r\sin\phi)-L|<\epsilon$. Let $$U(\phi_0)=\{(r,\phi)\in[0,\infty)\times\mathbb R\mid \|(r,\phi)-(0,\phi_0)\|<\delta(\phi_0)\}.$$ Then $U=\bigcup_{\phi_0\in\mathbb R}U(\phi_0)$ is an open neighborhood of $\{0\}\times\mathbb R$ in $[0,\infty)\times\mathbb R$ and for each $r\neq 0$ the condition $(r,\phi)\in U$ implies $\|f(r\cos\phi,r\sin\phi)- L\|<\epsilon$, because every such $(r,\phi)$ is an element of some $U(\phi_0)$ and therefore satisfies $\|(r,\phi)-(0,\phi_0)\|<\delta(\phi_0)$.
Finally, we show that 4 implies 1, thus concluding the circle of implications and establishing the desired equivalence. To do this, let $\epsilon>0$. By 4 and the fact that $f(r\cos\phi,r\sin\phi)$ is a $2\pi$-periodic function of $\phi$, there is a $\delta>0$ such that $0<r<\delta$ implies $|f(r\cos\phi,r\sin\phi)-L|<\epsilon$. Therefore, if $0<\|(x,y)\|<\delta$, we have $|f(x,y)-L|<\epsilon$, because $(x,y)=(r\cos\phi,r\sin\phi)$ for some $r\in(0,\delta)$ and $\phi\in\mathbb R$. This concludes the proof. $\qquad\square$
Remark. Note that the limit $(\star)$ is the limit of a two-variable function. Requiring that we have the single variable limit $$\lim_{r\to0}f(r\cos\phi_0,r\sin\phi_0)=L\tag{$\ast$}$$ for each $\phi_0\in\mathbb R$ is not (!) equivalent. In fact, this latter condition is not sufficient for the limit to exist as shown by the standard counterexample $f(x,y)=\frac{x^2y}{x^4+y^2}.$) The point is that the limit $(\star)$ takes into account points $(r,\phi)$ with different values of $\phi$ whereas in $(\ast)$ the value $\phi=\phi_0$ is fixed.
Added: here are the details regarding the part with the tube lemma, but I will do it without actually using the tube lemma, which would allow us to skip a few steps. (In fact, what follows basically includes a proof of the tube lemma in this special case.)
The definition of $(\diamond)$ mentions a neighborhood $U$ of $\{0\}\times\mathbb R$ in $[0,\infty)\times\mathbb R$. We wish to show that, if $g(r,\phi)$ is $2\pi$-periodic in $\phi$, we can always take a neighborhood of the form $V=[0,\delta)\times\mathbb R$ for some $\delta>0$, instead of this possibly more complicated neighborhood $U$.
Assume we have a neighborhood $U$ of $\{0\}\times\mathbb R$, as in the definition of $(\diamond)$, not necessarily of the form $[0,\delta)\times\mathbb R$. Using $U$, we are going to construct a neighborhood $V=[0,\delta)\times\mathbb R$ satisfying the same properties. Note that since $U$ is a neighborhood of every point $(0,\phi_0)\in\{0\}\times[0,2\pi]$, it contains a ball $$B((0,\phi_0),\eta(\phi_0))=\{(r,\phi)\in[0,\infty)\times\mathbb R\mid \|(r,\phi)-(0,\phi_0)\|<\eta(\phi_0)\}$$ of radius $\eta(\phi_0)>0$ around every such point. This ball contains a set of the form $$V(\phi_0)=[0,\delta(\phi_0))\times(\phi_0-\delta(\phi_0),\phi_0+\delta(\phi_0))$$ which still contains $(0,\phi_0)$. The sets $V(\phi_0)$ form a cover of the interval $\{0\}\times[0,2\pi]$, which is compact, so there is a finite subcover $V(\phi_1),V(\phi_2),\ldots, V(\phi_n)$, where $\phi_1,\phi_2,\ldots,\phi_n\in[0,2\pi]$. Now take $\delta=\min\{\delta(\phi_1),\delta(\phi_2),\ldots,\delta(\phi_n)\}$. Observe that $$[0,\delta)\times[0,2\pi]\subseteq V(\phi_1)\cup V(\phi_2)\cup\ldots\cup V(\phi_n)\subseteq U,$$ so $(r,\phi)\in[0,\delta)\times [0,2\pi]$ implies $|g(r,\phi)-L|<\epsilon$. But since $g$ is $2\pi$-periodic in $\phi$, this means that $(r,\phi)\in[0,\delta)\times\mathbb R$ also implies $|g(r,\phi)-L|<\epsilon$. Therefore $V=[0,\delta)\times\mathbb R$ has the desired properties.