Find all functions $ f : \mathbb R \to \mathbb R $ such that
$$ f \left( x ^ 2 y + f ( y ) \right) = x ^ 2 f ( y ) + f \big( f ( y ) \big) $$
for all $ x , y \in \mathbb R $.
It's straightforward to check that for any constant $ a \in \mathbb R $, the function $ f : \mathbb R \to \mathbb R $ defined by $ f ( x ) = a x $ for all $ x \in \mathbb R $ is a solution. I suspect that those are the only solutions.
Setting $ x = 1 $ and $ y = 0 $, we can see that $ f ( 0 ) = 0 $. Letting $ a = f ( 1 ) $ and defining $ g : \mathbb R \to \mathbb R $ with $ g ( x ) = f ( x ) – a x $ for all $ x \in \mathbb R $, we can see that $ g ( 0 ) = g ( 1 ) = 0 $, and the functional equation becomes
$$ g \Big( \left( x ^ 2 + a \right) y + g ( y ) \Big) = x ^ 2 g ( y ) + g \big( a y + g ( y ) \big) $$
for all $ x , y \in \mathbb R $. Setting $ y = 1 $, we find out that $ g ( x ) = g ( a ) $ for all $ x \ge a $. In case $ a \le 1 $, this directly gives $ g ( a ) = 0 $, as we have $ g ( 1 ) = 0 $. In case $ a > 0 $ one can set $ y = a $ and choose $ x $ large enough so that $ \left( x ^ 2 + a \right) y + g ( y ) \ge a $, and get $ x ^ 2 g ( a ) = g ( a ) – g \big( a ^ 2 + g ( a ) \big) $, and since this is true for all large enough $ x $, we must have $ g ( a ) = 0 $. So in any case, we have $ g ( x ) = 0 $ for all $ x \ge a $.
It remains to prove that $ g $ is zero everywhere, which will show that linear functions are the only possible solutions.
Source:
Number six on the list at the end of this page.
Best Answer
In fact, it's not true that linear functions are the only solutions. For any $ a , b \in \mathbb R _ { 0 + } $, the function $ f : \mathbb R \to \mathbb R $ defined with $$ f ( x ) = \begin {cases} a x & x \ge 0 \\ b x & x < 0 \end {cases} $$ is another solution. This can be verified easily, using the fact that $ y $, $ a y $, $ b y $, $ \left( x ^ 2 + a \right) y $ and $ \left( x ^ 2 + b \right) y $ will always have the same sign (in the nonstrict sense), because $ a , b \ge 0 $. Considering these solutions together with linear functions, one can show that there is no other solution, which we will prove.
We continue the argument put forward by the OP, except that we only prove $ g ( y ) = 0 $ for $ y > 0 $, as it may not be true for $ x < 0 $ in the case of the above solutions. Fixing $ y > 0 $, we can choose $ x $ large enough, namely $ x \ge \sqrt { \frac { | a - a y - g ( y ) | } y } $, so that we have $ \left( x ^ 2 + a \right) y + g ( y ) \ge a $, and hence $ g \Big( \left( x ^ 2 + a \right) y + g ( y ) \Big) = 0 $. Then we can use the functional equation for $ g $ to get $ x ^ 2 g ( y ) = - g \big( a y + g ( y ) \big) $, which shows that $ g ( y ) = 0 $, as this is true for all large enough $ x $.
For the negative points, we can take a similar path. Letting $ b = - f ( - 1 ) $ and defining $ h : \mathbb R \to \mathbb R $ with $ h ( x ) = f ( x ) - b x $ for all $ x \in \mathbb R $, we have $ h ( 0 ) = h ( - 1 ) = 0 $ and the functional equation $$ h \Big( \left( x ^ 2 + b \right) y + h ( y ) \Big) = x ^ 2 h ( y ) + h \big( b y + h ( y ) \big) $$ for all $ x , y \in \mathbb R $. Setting $ y = - 1 $, we get $ h ( x ) = h ( - b ) $ for all $ x \le - b $. In case $ b \le 1 $, we have $ - 1 \le - b $ and since $ h ( - 1 ) = 0 $, $ h ( - b ) = 0 $. In case $ b > 0 $, letting $ y = - b $ and choosing $ x $ large enough so that $ \left( x ^ 2 + b \right) y + h ( y ) \le - b $, we get $ x ^ 2 h ( - b ) = h ( - b ) - h \big( - b ^ 2 + h ( - b ) \big) $, which gives $ h ( - b ) = 0 $ since it's true for all large enough $ x $. So in any case, we have $ h ( x ) = 0 $ for all $ x \le - b $. Again, for any $ y < 0 $, choosing $ x $ large enough so that $ \left( x ^ 2 + b \right) y + h ( y ) \le - b $, we get $ x ^ 2 h ( y ) = - h \big( b y + h ( y ) \big) $, which shows that $ h ( y ) = 0 $, as that is true for all large enough $ x $.
So we know that $ g ( x ) = 0 $ for all $ x \ge \min ( a , 0 ) $ and that $ h ( x ) = 0 $ for all $ x \le \max ( - b , 0 ) $. Note that by definition of $ g $ and $ h $ we have $ g ( x ) - h ( x ) = ( a - b ) x $ for all $ x \in \mathbb R $. This implies that if $ a < 0 $, we can choose $ x $ so that $ a < x < 0 $, and thus have $ g ( x ) = h ( x ) = 0 $, which gives $ a = b $. Similarly, if $ b < 0 $, choosing $ x $ with $ 0 < x < - b $ we get $ a = b $. Therefore, if $ a $ is not equal to $ b $ then $ a , b \ge 0 $.
By definition of $ g $ and $ h $ and the previous results, we have $ f ( x ) = a x $ for $ x \ge 0 $ and $ f ( x ) = b x $ for $ x < 0 $. If $ f $ is not linear, then $ a $ is not equal to $ b $, and thus we must have $ a , b \ge 0 $. We already know that whatever the value of $ a $ and $ b $, we'll get a solution in such a case. Thus there is no solution other than linear ones and those of this form, and we're done.