Let $f\colon \Bbb Z\to \Bbb Z$ be a any function function with $$\tag0f(x-f(y)) = f(f(x)) - f(y) - 1$$
for all $x,y\in\Bbb Z$.
Letting $y=f(x)$ we find $$f(x-f(f(x)))=-1. $$
So for $a=-f(f(0))$ we have $f(a)=-1$. Then with $y=a$, $$\tag1f(x+1)=f(f(x)) $$
Then $(0)$ becomes
$$\tag2 f(x-f(y))=f(x+1)-f(y)-1 $$
Or with $g(x):=f(x)+1$ (and $x\leftarrow x-1$)
$$\tag3g(x-g(y))=g(x)-g(y)$$
From $(3)$ we see that the image of $g$ is a subgroup of $\Bbb Z$, hence it is either $\{0\}$ (in which case $f(x)=-1$), or $c\Bbb Z$ for some $c\ge 1$.
In that case, for $n\in\Bbb Z$ we find $y$ with $g(y)=nc$ and so have $g(x+ nc)=g(x)+nc$. Thus $g$ is determined by the values $g(0),\ldots, g(c-1)$. On the other hand, these values can indeed be chosen freely. In other words:
Claim 1. Let $c\in\Bbb N$ and $b_0,\ldots, b_{c-1}\in\Bbb Z$. Then the function $g$ given by
$$ g(x)= (n+b_r)c$$
where $x=nc+r$, $0\le r<c$
is a solution to $(3)$, and all non-zero solutions of $(3)$ are obtained this way.
Proof.
Let $x=nc+r$, $y=mc+s$ with $0\le r,s<c$. Then
$$\begin{align}g(x-g(y))&=g(nc+r-(m+b_s)c)\\
&=g((n-m-b_s)c+r)\\
&=(n-m-b_s+b_{r})c\\
&=(n+b_r)c-(m+b_s)c\\&=g(x)-g(y)\end{align}$$
That all non-zero solutions are of this form has been shown above. $\square$
Then the solutions $f$ of $(2)$ (apart from $f(x)=-1$) are precisely those of the form $f(x)=g(x)-1$ with $g$ as in Claim 1.
Such $f$ is a solution to the original $(0)$ if and only if we additionally have $(1)$ for all $x$.
Note that for $x=nc+r$, $0\le r<c$, we have $f(x)=g(x)-1=(n+b_r)c-1=(n+b_r-1)c+c-1$ so that $$f(f(x))=(n+b_r-1+b_{c-1})c-1.$$
On the other hand,
$$f(x+1)=g(x+1)-1=\begin{cases}(n+b_{r+1})c-1&\text{if }r<c-1\\(n+1+b_0)c-1&\text{if }r=c-1\end{cases}$$
We conclude that $b_{r+1}=b_r+b_{c-1}-1$ for $0\le r<c-1$, and that $2b_{c-1}-1=b_0+1$. From the first we see that $b_r=b_0+rb_{c-1}-r$, so
$$\begin{align}b_{c-1}&=b_0+1+(c-1)b_{c-1}-c\\
&=2b_{c-1}-1+(c-1)b_{c-1}-c\\
&=(c+1)b_{c-1}-c-1\end{align}$$ and finally
$$b_{c-1} = \frac {c+1}c.$$
This is an integer only for $c=1$ and in that case we arrive at $b_0=2$
Thus the only solutions to $(0)$ apart from $f(x)=-1$ is
$$f(x)=x+1.$$
First of all, note that the problem can be stated in the following equivalent form.
Find all functions $ f : \mathbb R \to \mathbb R $ such that
$$ f \left( - f ( x ) - f ^ 2 ( y ) \right) ^ 3 + 3 x y \left( f ( x ) + f ^ 2 ( y ) \right) + x f ( x ) ^ 2 + y ^ 2 f ( y ) = 0 \tag 0 \label 0 $$
for all $ x , y \in \mathbb R $. Here, $ f ^ 2 ( y ) $ means $ f \bigl( f ( y ) \bigr) $, and $ f ( x ) ^ 2 $ means $ f ( x ) \cdot f ( x ) $.
It's straightforward to verify that the constant zero function, the identity function and the additive inversion function all satisfy the required condition. We prove that these are the only solutions.
Letting $ k = f ( 0 ) $ and putting $ x = y = 0 $ in \eqref{0}, we get $ f \bigl( - k - f ( k ) \bigr) = 0 $. Then, plugging $ x = - k - f ( k ) $ and $ y = 0 $ in \eqref{0} we can see that $ f \bigl( - f ( k ) \bigr) = 0 $. Now, set $ x = - f ( k ) $ in \eqref{0} and consider the equation once with $ y = - k - f ( k ) $ and another time with $ y = - f ( k ) $. Comparing the two results, you can conclude $ k ^ 2 f ( k ) = 0 $, which together with $ f \bigl( - f ( k ) \bigr) = 0 $ implies $ k = 0 $.
Putting $ y = 0 $ in \eqref{0} gives
$$ f \bigl( - f ( x ) \bigr) ^ 3 = - x f ( x ) ^ 2 \tag 1 \label 1 $$
for all $ x \in \mathbb R $, while plugging $ x = 0 $ in \eqref{0} yields
$$ f \left( - f ^ 2 ( y ) \right) ^ 3 = - y ^ 2 f ( y ) \tag 2 \label 2 $$
for all $ y \in \mathbb R $. Letting $ x = f ( y ) $ in \eqref{1}, we get $ f \left( - f ^ 2 ( y ) \right) ^ 3 = - f ( y ) f ^ 2 ( y ) ^ 2 $, which together with \eqref{2} shows that for any $ y \in \mathbb R \setminus \{ 0 \} $ with $ f ( y ) \ne 0 $ we have $ f ^ 2 ( y ) ^ 2 = y ^ 2 $. But also note that if $ y , f ( y ) \ne 0 $ and $ f ^ 2 ( y ) = - y $, then \eqref{2} gives $ f ( y ) \in \{ - y , y \} $; $ f ( y ) = y $ cannot happen since it implies $ f ^ 2 ( y ) = y $ (which contradicts $ f ^ 2 ( y ) = - y $ as $ y \ne 0 $), and $ f ( y ) = - y $ is not possible because it implies $ f ( - y ) = f ^ 2 ( y ) = - y $, which then substituting $ - y $ for $ y $ in \eqref{2} implies $ f ( y ) = y $ (contradicting $ f ( y ) = - y $ as $ y \ne 0 $). Therefore, the case $ f ^ 2 ( y ) = - y $ is ruled out, and we must have
$$ f ( y ) \ne 0 \implies f ^ 2 ( y ) = y \tag 3 \label 3 $$
for all $ y \in \mathbb R \setminus \{ 0 \} $. Now, assume that there exists $ y _ 0 \in \mathbb R \setminus \{ 0 \} $ with $ f ( y _ 0 ) \ne 0 $. Putting $ y = y _ 0 $ in \eqref{2} and using \eqref{3} we have $ f ( - y _ 0 ) ^ 3 = - y _ 0 ^ 2 f ( y _ 0 ) $. Using this and \eqref{3} and setting $ y = y _ 0 $ in \eqref{0}, we can see that if $ f ( x ) = 0 $ for some $ x \in \mathbb R $, then we must have $ x = 0 $. Hence, using \eqref{3} we can conclude $ f ^ 2 ( y ) = y $ for all $ y \in \mathbb R $. Consequently, substituting $ f ( x ) $ for $ x $ in \eqref{0}, we get
$$ f ( - x - y ) ^ 3 + 3 y ( x + y ) f ( x ) + x ^ 2 f ( x ) + y ^ 2 f ( y ) = 0 \tag 4 \label 4 $$
for all $ x , y \in \mathbb R $. Interchanging $ x $ and $ y $ in \eqref{4} and comparing with \eqref{4} itself, it's straightforward to see that
$$ y f ( x ) = x f ( y ) $$
for all $ x , y \in \mathbb R $. In particular, this gives $ f ( x ) = f ( 1 ) x $ for all $ x \in \mathbb R $, and as $ f ^ 2 ( 1 ) = 1 $, we have $ f ( 1 ) \in \{ - 1 , 1 \} $. Therefore, we've proven that if $ f $ differs from the constant zero function, we must either have $ f ( x ) = x $ for all $ x \in \mathbb R $ or $ f ( x ) = - x $ for all $ x \in \mathbb R $, which proves what was claimed.
Best Answer
This is not correct. You can not let $u=f(x)$ since you haven't prove that $f(x)$ is surjective.
Here is a hint: use $f(f(x))=f(x)+f(0)$ to change the LHS or the original functional equation so you have $f(x+y)+f(0)=f(x)+f(y)$. This really looks like Cauchy's functional equation. Can you keep going?