Note that the d.e is of the form $M(x,y)\text{d}x+N(x,y) \text{d}y$ with
$$ M(x,y)=2xy^2+\cos (x),~N(x,y)=2x^2y+\sin (y) $$
and the equation is exact if and only if $\displaystyle\frac {\partial M}{\partial y}=\frac{\partial N}{\partial x}.$
Now lets check if the differential equation is indeed exact.
$\displaystyle\frac{\partial M}{\partial y}=\frac{\partial}{\partial y}\left(2xy^{2}+\cos x\right)=4xy=\frac{\partial}{\partial x}\left(2x^{2}y+\sin y\right)=\frac{dN}{\partial x}$.
So the equation is exact.
The general solution is of the form $f\left(x,y\right)=C$ and it's given by
$$
f\left(x,y\right)=\int M(x,y)\text{d}x =\int(2xy^{2}+\cos x)\text{d}x=x^{2}y^{2}+\sin(x)+ g(y).$$
To find $g\left(y\right)$ differentiate $f\left(x,y\right)$ partially
with respect to $'y'$ and compare with $N\left(x,y\right).$
That is:
$f_{y}\left(x,y\right)=\dfrac{\partial}{dy}\left(x^{2}y^{2}+\sin x+g(y)\right)=2x^{2}y+g'\left(y\right).$
Comparing with $N\left(x,y\right)$, we find $g'\left(y\right)=\sin y,$
which implies that $g\left(y\right)=-\cos y+K$.
Hence, the general solution is $ x^2y^2+\sin x-\cos y=C.\quad\quad\Box$
Added
What happens if $M(x,y)\text{d}x+N(x,y)\text{d}y=0$ is not exact? Then there exist a function $u(x,y)$ such that $$[u(x,y)M(x,y)]\text{d}x+[u(x,y)N(x,y)]\text{d}y=0$$ is exact. The function $u$ is called an integrating factor. Furthermore, If $$\frac{M_y-N_x}{N}$$ is a function of $x$ only, say, $v(x)$, then $u(x,y)=u(x)=e^{\int v(x)\text{d}x}$.
On the other hand if $$\frac{M_y-N_x}{-M}$$ is a function of $y$ only, say, $w(y)$, then $u(x,y)=u(y)=e^{\int w(y)\text{d}y}$.
As an example, consider $(3xy-y^2)\text{d}x+(x^2-xy)\text{d}y=0$. Clearly, this is not exact. But $\dfrac{M_y-N_x}{N}=\dfrac{1}{x}=v(x)$, a function of $x$ only. So our integrating factor becomes $$u(x)=e^{\int \frac{1}{x}\text{d}x}=e^{\ln |x|}=|x|.$$
Verify that $$(3x^2y-xy^2)\text{d}x+(x^3-x^2y)\text{d}y=0$$ is now exact!
We now proceed as before to find the general solution.
Your original differential equation
$$
(x^3+xy^2-y)\,dx+(y^3+yx^2+x)\,dy=0
$$
is not exact. What one might do is to multiply it by a function (called integrating factor) $\mu(x,y)$ to make
$$
\mu(x,y)(x^3+xy^2-y)\,dx+\mu(x,y)(y^3+yx^2+x)\,dy=0
$$
exact.
In general, if one starts with
$$
P(x,y)\,dx+Q(x,y)\,dy=0,
$$
then the differential equation
$$
\mu(x,y)\bigl(P(x,y)\,dx+Q(x,y)\,dy\bigr)=0
$$
is exact if (check this) $\mu$ satisfies
$$
P\mu'_y-Q\mu'_x+\mu\bigl(P'_y-Q'_x\bigr)=0.\tag{*}
$$
You do not need to find the general solution to this partial differential equation, it suffices to find a particular solution. In your case, one solution to $(^*)$ is given by
$$
\mu(x,y)=\frac{1}{x^2+y^2}.
$$
This is why your differential equation becomes exact after multiplying with that $\mu$.
Best Answer
Like you said: $$ \frac{\partial F(x,y)}{\partial x} = p(x,y), \quad \text{and} \frac{\partial F(x,y)}{\partial y} = q(x,y), $$ then $F(x,y) = C$ solves $$ p(x,y) + q(x,y)\frac{dy}{dx} = 0, $$ basically we take derivative w.r.t. $x$ in $F(x,y) = C$.
Now your equation reads: $$ \frac{\partial F(x,y)}{\partial x} = 2y^2+6xy-x^2 \implies F(x,y) = \int ( 2y^2+6xy-x^2) dx + \color{red}{H(y)}, $$ where $H(y)$ is the key, because a function purely based on $y$ is lost when we take $\partial_x$. Then $$ F(x,y) = 2y^2x +3x^2y - \frac{x^3}{3}+ H(y).\tag{1} $$ Here we can use the second condition: $$ \frac{\partial F(x,y)}{\partial y} = y^2 + 4xy + 3x^2, $$ plugging (1) into above: $$ \frac{\partial }{\partial y}\left(2y^2x +3x^2y - \frac{x^3}{3}+ H(y)\right) = y^2 + 4xy + 3x^2, $$ for first three terms, take partial derivative w.r.t. $y$. For $H(y)$, which is a function only of $y$-variable, take $\partial/\partial y$ is the same as $d/dy$, hence this gives you (notice some terms get canceled): $$ \frac{d }{d y} H(y) = \ldots $$ integrating both w.r.t. $y$ as if you are integrating only a one independent variable ODE, then plugging back to (1) you will get your $F(x,y)$.