First of all, note that the problem can be stated in the following equivalent form.
Find all functions $ f : \mathbb R \to \mathbb R $ such that
$$ f \left( - f ( x ) - f ^ 2 ( y ) \right) ^ 3 + 3 x y \left( f ( x ) + f ^ 2 ( y ) \right) + x f ( x ) ^ 2 + y ^ 2 f ( y ) = 0 \tag 0 \label 0 $$
for all $ x , y \in \mathbb R $. Here, $ f ^ 2 ( y ) $ means $ f \bigl( f ( y ) \bigr) $, and $ f ( x ) ^ 2 $ means $ f ( x ) \cdot f ( x ) $.
It's straightforward to verify that the constant zero function, the identity function and the additive inversion function all satisfy the required condition. We prove that these are the only solutions.
Letting $ k = f ( 0 ) $ and putting $ x = y = 0 $ in \eqref{0}, we get $ f \bigl( - k - f ( k ) \bigr) = 0 $. Then, plugging $ x = - k - f ( k ) $ and $ y = 0 $ in \eqref{0} we can see that $ f \bigl( - f ( k ) \bigr) = 0 $. Now, set $ x = - f ( k ) $ in \eqref{0} and consider the equation once with $ y = - k - f ( k ) $ and another time with $ y = - f ( k ) $. Comparing the two results, you can conclude $ k ^ 2 f ( k ) = 0 $, which together with $ f \bigl( - f ( k ) \bigr) = 0 $ implies $ k = 0 $.
Putting $ y = 0 $ in \eqref{0} gives
$$ f \bigl( - f ( x ) \bigr) ^ 3 = - x f ( x ) ^ 2 \tag 1 \label 1 $$
for all $ x \in \mathbb R $, while plugging $ x = 0 $ in \eqref{0} yields
$$ f \left( - f ^ 2 ( y ) \right) ^ 3 = - y ^ 2 f ( y ) \tag 2 \label 2 $$
for all $ y \in \mathbb R $. Letting $ x = f ( y ) $ in \eqref{1}, we get $ f \left( - f ^ 2 ( y ) \right) ^ 3 = - f ( y ) f ^ 2 ( y ) ^ 2 $, which together with \eqref{2} shows that for any $ y \in \mathbb R \setminus \{ 0 \} $ with $ f ( y ) \ne 0 $ we have $ f ^ 2 ( y ) ^ 2 = y ^ 2 $. But also note that if $ y , f ( y ) \ne 0 $ and $ f ^ 2 ( y ) = - y $, then \eqref{2} gives $ f ( y ) \in \{ - y , y \} $; $ f ( y ) = y $ cannot happen since it implies $ f ^ 2 ( y ) = y $ (which contradicts $ f ^ 2 ( y ) = - y $ as $ y \ne 0 $), and $ f ( y ) = - y $ is not possible because it implies $ f ( - y ) = f ^ 2 ( y ) = - y $, which then substituting $ - y $ for $ y $ in \eqref{2} implies $ f ( y ) = y $ (contradicting $ f ( y ) = - y $ as $ y \ne 0 $). Therefore, the case $ f ^ 2 ( y ) = - y $ is ruled out, and we must have
$$ f ( y ) \ne 0 \implies f ^ 2 ( y ) = y \tag 3 \label 3 $$
for all $ y \in \mathbb R \setminus \{ 0 \} $. Now, assume that there exists $ y _ 0 \in \mathbb R \setminus \{ 0 \} $ with $ f ( y _ 0 ) \ne 0 $. Putting $ y = y _ 0 $ in \eqref{2} and using \eqref{3} we have $ f ( - y _ 0 ) ^ 3 = - y _ 0 ^ 2 f ( y _ 0 ) $. Using this and \eqref{3} and setting $ y = y _ 0 $ in \eqref{0}, we can see that if $ f ( x ) = 0 $ for some $ x \in \mathbb R $, then we must have $ x = 0 $. Hence, using \eqref{3} we can conclude $ f ^ 2 ( y ) = y $ for all $ y \in \mathbb R $. Consequently, substituting $ f ( x ) $ for $ x $ in \eqref{0}, we get
$$ f ( - x - y ) ^ 3 + 3 y ( x + y ) f ( x ) + x ^ 2 f ( x ) + y ^ 2 f ( y ) = 0 \tag 4 \label 4 $$
for all $ x , y \in \mathbb R $. Interchanging $ x $ and $ y $ in \eqref{4} and comparing with \eqref{4} itself, it's straightforward to see that
$$ y f ( x ) = x f ( y ) $$
for all $ x , y \in \mathbb R $. In particular, this gives $ f ( x ) = f ( 1 ) x $ for all $ x \in \mathbb R $, and as $ f ^ 2 ( 1 ) = 1 $, we have $ f ( 1 ) \in \{ - 1 , 1 \} $. Therefore, we've proven that if $ f $ differs from the constant zero function, we must either have $ f ( x ) = x $ for all $ x \in \mathbb R $ or $ f ( x ) = - x $ for all $ x \in \mathbb R $, which proves what was claimed.
Best Answer
It's straightforward to verify that for any $ c \in \mathbb R $ and any additive idempotent $ g : \mathbb R \to \mathbb R $ satisfying $ g ( c ) = c $, the function $ f : \mathbb R \to \mathbb R $ defined with $ f ( x ) = g ( x ) + c $ for all $ x \in \mathbb R $, satisfies $$ f \bigl ( f ( x ) + y + z \bigr ) = f ( x ) + f ( y ) - f ( - z ) + f ( 0 ) \tag 0 \label 0 $$ for all $ x , y , z \in \mathbb R $. To prove that these are the only solutions, consider an $ f : \mathbb R \to \mathbb R $ satisfying \eqref{0} for all $ x , y , z \in \mathbb R $, and let $ c = f ( 0 ) $. Substituting $ y + z $ for $ y $ and $ 0 $ for $ z $ in \eqref{0}, you get $$ f \bigl ( f ( x ) + y + z \bigr ) = f ( x ) + f ( y + z ) $$ for all $ x , y , z \in \mathbb R $, which comparing with \eqref{0} itself, yields $$ f ( y + z ) = f ( y ) - f ( - z ) + c \tag 1 \label 1 $$ for all $ y , z \in \mathbb R $. Interchanging $ y $ and $ z $ in \eqref{1} and comparing with \eqref{1} itself, you have $$ f ( y ) - f ( - z ) = f ( z ) - f ( - y ) $$ for all $ y , z \in \mathbb R $, which in particular for $ z = 0 $ implies $$ f ( - y ) = - f ( y ) + 2 c \tag 2 \label 2 $$ for all $ y \in \mathbb R $. Define $ g : \mathbb R \to \mathbb R $ with $ g ( x ) = f ( x ) - c $ for all $ x \in \mathbb R $. Then you have $ g ( 0 ) = 0 $, while \eqref{2} implies $$ g ( - y ) = - g ( y ) \tag 3 \label 3 $$ for all $ y \in \mathbb R $. By \eqref{1} and \eqref{3}, you get $$ g ( y + z ) = g ( y ) + g ( z ) \tag 4 \label 4 $$ for all $ y , z \in \mathbb R $; i.e. $ g $ is additive. \eqref{0} becomes $$ g \bigl ( g ( x ) + y + z + c \bigr ) + c = g ( x ) + c + g ( y ) + c - g ( - z ) - c + c \text , $$ which using \eqref{3} and \eqref{4} implies $$ g \bigl ( g ( x ) \bigr ) + g ( y ) + g ( z ) + g ( c ) = g ( x ) + g ( y ) + g ( z ) + c \text , $$ for all $ x , y , z \in \mathbb R $, or $$ g \bigl ( g ( x ) \bigr ) + g ( c ) = g ( x ) + c \tag 5 \label 5 $$ for all $ x \in \mathbb R $. Setting $ x = 0 $ in \eqref{5} you get $ g ( c ) = c $, and therefore \eqref{5} imples $$ g \bigl ( g ( x ) \bigr ) = g ( x ) $$ for all $ x \in \mathbb R $; i.e. $ g $ is idempotent, and we're done.
The characterization given above is the best you can hope for. Without further assumptions on $ f $, there are uncountably many wild solutions. Take a look at "Overview of basic facts about Cauchy functional equation" to get more information about the matter. In case you have regularity assumptions (like continuity, local boundedness or measurability) on $ f $, $ g $ will be regular, too, and thus it must be of the form $ g ( x ) = a x $ for some constant $ a \in \mathbb R $ and all $ x \in \mathbb R $. By idempotency of $ g $, you get $ a ^ 2 = a $, and therefore $ a \in \{ 0 , 1 \} $. As you also have $ g ( c ) = c $, $ a = 0 $ is only possible if $ c = 0 $. Therefore, the only regular solutions of the problem are those of the form $ f ( x ) = x + c $ for some constant $ c \in \mathbb R $, and the constant zero function $ f ( x ) = 0 $.
To characterize all the possible $ g $ above when no further assumptions hold, consider a Hamel basis $ \mathcal H $ of $ \mathbb R $ over $ \mathbb Q $, which contains $ c $ in case $ c \ne 0 $. Partition $ \mathcal H $ into two subsets $ \mathcal X $ and $ \mathcal Y $, such that $ \mathcal Y $ contains $ c $ if $ c \ne 0 $. Consider any $ h : \mathcal X \to \operatorname {span} _ { \mathbb Q } ( \mathcal Y ) $, and define $ g $ to be the unique $ \mathbb Q $-linear function on $ \mathbb R $ which simultaneously extends $ h $ and $ \operatorname {Id} _ { \mathcal Y } $. Then $ g $ will be and additive idempotent function on $ \mathbb R $ with $ g ( c ) = c $. Conversely, given any such $ g $, the set of fixed point of $ g $ will be a $ \mathbb Q $-subspace of $ \mathbb R $. Considering a Hamel basis $ \mathcal Y $ of this set over $ \mathbb Q $ (which can be chosen so that it contains $ c $ if $ c \ne 0 $) and extending it to a Hamel basis of $ \mathbb R $ over $ \mathbb Q $, you can see that $ g $ is of the previously mentioned form.