First of all:
$\frac{\partial f(x,y)}{\partial x} = \frac{2xy(x^4+y^2)-4x^3(x^2y)}{(x^4+y^2)^2}$
$\frac{\partial f(x,y)}{\partial y} = \frac{x^2(x^4+y^2)-2y(x^2y)}{(x^4+y^2)^2}$
and neither of the partial derivatives $\frac{\partial f(x,y)}{\partial x}$, $\frac{\partial f(x,y)}{\partial y}$ exist at $f(0,0)$
This is not right. The partial derivatives are particular directional derivatives (using the vectors $e_1=(1,0)$ and $e_2=(0,1)$), and you have to show that it exist (by definition below).
(As you wrote) Given a vector $u=(u_1,u_2)$ with $\|u\|=1$ we define the directional derivative of a function $f$ at a point $p=(p_1,p_2)$ by $$\dfrac{\partial f}{\partial u}(p) = \lim_{t\to 0} \dfrac{f(p+tu)-f(p)}{t} = \lim_{t \to 0} \dfrac{f(p_1+tu_1,p_2+tu_2) - f(p_1,p_2)}{t}$$ if the limit exist.
You have to show that this limit exist for every $u\in \Bbb{R}^2$ such that $\|u\|=1$.
Edit: To show that $f$ isn't differentiable at $(0,0)$, you can calculate the limit through different paths and conclude that the limits hasn't the same value, like Svetoslav said in comments.
Let $f(x,y)$ be defined as
$$f(x,y)=\begin{cases}\frac{x^2y}{x^2+y^2}, &(x,y)\ne(0,0)\\\\0,& (x,y)=(0,0)\end{cases}$$
The directional derivative of $f(x,y)$ along the unit vector $(a,b)$ (i.e. $a^2+b^2=1$) at $(x,y)=(0,0)$ is given by the limit
$$\begin{align}
\lim_{h\to 0}\frac{f(ah,bh)-f(0,0)}{h}&=2\lim_{h\to 0} \frac{\frac{a^2h^2bh}{a^2h^2+b^2h^2}}{h}\\\\
&=\frac{2a^2b}{a^2+b^2}\\\\
&=2a^2b
\end{align}$$
So, the directional derivatives exist in all directions at the origin.
It is easy to show that both first partial derivatives vanish at the origin. It remains to show that the limit
$$\lim_{(h,k)\to (0,0)}\frac{f(h,k)-f(0,0)-f_1(0,0)h-f_2(0,0)k}{\sqrt{h^2+k^2}}$$
fails to exist. Proceeding we see that
$$\begin{align}
\frac{f(h,k)-f(0,0)-f_1(0,0)h-f_2(0,0)k}{\sqrt{h^2+k^2}}&=\frac{2h^2k}{(h^2+k^2)^{3/2}}\\\\
&=2\frac{h^2}{h^2+k^2}\frac{k}{\sqrt{h^2+k^2}}\tag1
\end{align}$$
Now, if $h=k$, the limit of $(1)$ is $2^{-1/2}$ while if $h=0$ or $k=0$ the limit is $0$. Inasmuch as the limit of $(1)$ fails to exist, we conclude that $f$ is not differentiable.
Best Answer
By the mean value theorem we see that
$$| \arctan t| \le |t|$$
for all $t$. Hence
$$ |\frac{\arctan(\cos(t)\sin(t)h^2)}{h}| \le |h|.$$
Conclusion ?