Find all functions $ f : \mathbb R \to \mathbb R $ such that
$$ f \left( x ^ 2 + f ( y ) \right) = f \big( f ( x ) \big) + f \big( f ( y ) \big) + x ^ 2 f ( y ) + y ^ 2 f ( x ) \tag 0 \label 0 $$
for all $ x , y \in \mathbb R $.
It's straightforward to check that the constant zero function and the squaring function are both solutions. I suspect that those are the only ones.
Plugging $ x = y = 0 $ in \eqref{0} gives $ f \big( f ( 0 ) \big) = 0 $, and the putting $ x = 0 $ and $ y = 1 $ in \eqref{0} shows that $ f ( 0 ) = 0 $. Then, setting $ y = 0 $ in \eqref{0} gives
$$ f \left( x ^ 2 \right) = f \big( f ( x ) \big) \tag 1 \label 1 $$
for all $ x \in \mathbb R $. We can use \eqref{1} to rewrite \eqref{0} as
$$ f \left( x ^ 2 + f ( y ) \right) = f \left( x ^ 2 \right) + f \left( y ^ 2 \right) + x ^ 2 f ( y ) + y ^ 2 f ( x ) \tag 2 \label 2 $$
for all $ x , y \in \mathbb R $. Substituting $ – x $ for $ x $ in \eqref{2} and comparing the result with \eqref{2} itself (for example when $ y = 1 $), we see that $ f $ is an even function; i.e.
$$ f ( – x ) = f ( x ) \tag 3 \label 3 $$
for all $ x \in \mathbb R $. Finally, exchanging $ x $ and $ y $ in \eqref{0}, we get
$$ f \left( x ^ 2 + f ( y ) \right) = f \left( y ^ 2 + f ( x ) \right) \tag 4 \label 4 $$
for all $ x , y \in \mathbb R $.
This is as far as I could go in this direction. Some messy calculations show that the only real analytic solutions are in fact those of the form $ f ( x ) = 0 $ and $ f ( x ) = x ^ 2 $, but this stil leaves open the possibility of another (non-analytic) solution.
Source:
Number eight on the list at the end of this page.
Best Answer
Suppose $f$ is not the constant zero function.
If $f(y_1)=f(y_2)$ for some $y_1, y_2$ then letting $y:=y_1$, $y:=y_2$ and $x$ such that $f(x)\neq 0$ we obtain $y_1=\pm y_2$.
Then it follows from $f(f(x))=f(x^2)$ that for all $x$ either $f(x)=x^2$ or $f(x)=-x^2$.
If $f(x)=-x^2$ for some $x$ then letting $y:=x$ we get $f(x^2)=x^4$. In particular, $f(1)=1$.
Letting $y:=1$ we obtain $f(x^2+1)=f(x^2)+1+x^2+f(x)$ (*). If $x\neq 0$ satisfies $f(x)=-x^2$ then we have two possibilities:
(i) $f(x^2+1)=(x^2+1)^2$. Then (*) reduces to $(x^2+1)^2=x^4+1$, a contradiction.
(ii) $f(x^2+1)=-(x^2+1)^2$. Then (*) reduces to $-(x^2+1)^2=x^4+1$, again a contradiction.
Hence $f(x)=x^2$ for all $x\neq 0$. Therefore $f(x)=x^2$ for all $x$ which clearly works.