Suppose $f:\Bbb R\to \Bbb R$ satisfies
$$f\big(f(x)-x+y^2\big)=yf(y)\tag{1}$$
for all $x,y\in \Bbb R$. Plugging in $x,y=1$ in $(1)$, we get
$$f\big(f(1)\big)=f(1).$$
Hence, substituting $x=f(1)$ in $(1)$, we obtain
$$f(y^2)=yf(y).\tag{2}$$
This shows that $f(0)=0$ and for $y\ne 0$,
$$-yf(-y)=f\big((-y)^2\big)=f(y^2)=yf(y),$$
so $f(-y)=-f(y)$. Hence $f$ is an odd function.
From $(1)$ and $(2)$, we get
$$f\big(f(x)-x+y^2\big)=f(y^2)$$
for all $x,y\in\mathbb{R}$. Substitute $-x$ for $x$ in the last equation and using the fact that $f$ is odd, we get
$$f\big(f(x)-x-y^2\big)=f(-y^2)$$
for all $x,y\in\mathbb{R}$. Hence,
$$f\big(f(x)-x+y)=f(y)\tag{3}$$
for all $x,y\in\mathbb{R}$.
Define $P$ to be the additive subgroup of $\mathbb{R}$ generated by $\big\{f(x)-x\big|x\in\Bbb R\big\}$. Then, we see that for any $z\in\mathbb{R}$ and $p\in P$, we have
$$f(z+p)=f(z).\tag{4}$$
Consequently $f$ is an odd periodic function which is invariant under translation by elements of $P$.
We claim that $P=f^{-1}(0)$. First if $p\in P$, then $f(p)=f(p+0)=f(0)=0$, so $p\in f^{-1}(0)$. Conversely, suppose that $p\in f^{-1}(0)$, then with $x=p$ in $(3)$, we have
$$f(-p+y)=f\big(f(p)-p+y\big)=f(y)$$
so that $p\in P$.
For each $p\in P$, we see that
$$f\left(\left(\frac{1+p}{2}\right)^2\right)=f\left(\left(\frac{1-p}{2}\right)^2+p\right)=f\left(\left(\frac{1-p}{2}\right)^2\right)$$
From $(2)$, we have
$$\frac{1+p}{2}f\left(\frac{1+p}{2}\right)=\frac{1-p}{2}f\left(\frac{1-p}{2}\right).$$
However
$$f\left(\frac{1+p}{2}\right)=f\left(\frac{1-p}{2}+p\right)=f\left(\frac{1-p}{2}\right).$$
This means
$$f\left(\frac{1+p}{2}\right)=0$$
for all $p\in P$ such that $p\ne 0$. Hence, for $p\in P\setminus\{0\}$, $\frac{1+p}{2}\in P$, so $1+p \in P$, making $1\in P$. This shows that either $P=\{0\}$ or $\mathbb{Z}\left[\frac12\right]\subseteq P$.
If $P=\{0\}$, then $f(x)-x=0$ for all $x\in \Bbb R$. Therefore, $f(x)=x$ for every $x\in \Bbb R$. This yields one solution of $(1)$. From now on $P\neq \{0\}$. Thus $\mathbb{Z}\left[\frac12\right]\subseteq P$. We want to show that $P=\Bbb R$, so that $f(x)=0$ for every $x\in \Bbb R$, and this is another solution of $(1)$.
For an arbitrary $y\in\mathbb{R}$, we see that
$$f\left(\left(y+\frac{1}{2}\right)^2\right)=\left(y+\frac{1}{2}\right)f\left(y+\frac{1}{2}\right)=\left(y+\frac{1}{2}\right)f(y),$$
since $1/2\in P$. That is
$$f\left(\left(y+\frac12\right)^2\right)=\frac{1}{2}f(y)+yf(y)=\frac12f(y)+f(y^2).$$
Consequently
\begin{align}f\left(\left(y+\frac12\right)^2\right)-\left(y+\frac12\right)^2&=\frac{1}{2}f(y)+f(y^2)-\left(y+\frac12\right)^2\\&=\left(\frac{1}{2}f(y)-y\right)+\left(f(y^2)-y^2\right)+\frac14.\end{align}
Because $f\left(\left(y+\frac12\right)^2\right)-\left(y+\frac12\right)^2$, $f(y^2)-y^2$, and $\frac14$ are all elements of $P$, we conclude that
$$\frac{1}{2}f(y)-y\in P.$$
Therefore,
$$\big(f(y)-y\big)-y=2\left(\frac{1}{2}f(y)-y\right)\in P.$$
As $f(y)-y\in P$, we get $y\in P$.
Suppose that $f:\mathbb{R}\to\mathbb{R}_{\geq 0}$ is a function such that $$f(x^2+y^2)=f(x^2-y^2)+f(2xy)\,,\tag{*}$$
for all $x,y\in\mathbb{R}$. For each $x,y\in\mathbb{R}$, let $P(x,y)$ denote the statement (*).
From $P(0,0)$, we get $f(0)=0$. Thus, $P(0,y)$ yields $$f(+y^2)=f(-y^2)$$ for all $y\in\mathbb{R}$. Therefore, $f$ is an even function. Let $F:\mathbb{R}_{\geq 0}\to\mathbb{R}$ be the function defined by
$$F(t):=f(\sqrt{t})$$
for all $t\geq 0$.
Now, since $f$ is an even function,
$$\begin{align}F\big((x^2+y^2)^2\big)&=f(x^2+y^2)=f(x^2-y^2)+f(2xy)\\&=f\big(|x^2-y^2|\big)+f\big(2|xy|\big)\\&=F\big((x^2-y^2)^2\big)+F\big((2xy)^2\big)\end{align}$$
for all $x,y\in\mathbb{R}$. This shows that $$F(u+v)=F(u)+F(v)\tag{#}$$
for all $u,v\geq 0$. (Note that, for $u,v\geq 0$, there are $x,y\in\mathbb{R}$ such that $u=(x^2-y^2)^2$ and $v=(2xy)^2$.) Thus, $F$ satisfies Cauchy's functional equation and $F$ is a nonnegative function. Therefore, $F$ is a nondecreasing function. Ergo, there exists $k\geq 0$ such that
$$F(t)=kt$$
for all $t\in\mathbb{R}_{\geq 0}$.
Since $f(x)=f\big(|x|\big)=F(x^2)$ for all $x\in\mathbb{R}$, we conclude that
$$f(x)=kx^2$$
for all $x\in\mathbb{R}$ (where $k\geq 0$ is a constant). It is easy to see that all functions $f$ of this form obey (*).
Remark. For a function $F:\mathbb{R}_{\geq 0}\to\mathbb{R}$ such that $F$ satisfies Cauchy's functional equation (#), we can conclude that there exists a constant $k$ such that $F(t)=kt$ for all $t\geq 0$ if at least one of the following properties are known to be true:
- $F$ is continuous at any point,
- $F$ is continuous at one point,
- $F$ is bounded on any bounded interval (with nonempty interior),
- $F$ is bounded on one bounded interval (with nonempty interior),
- $F$ is monotonic, or
- $F$ is monotonic on one interval (with nonempty interior).
Without these properties, there are noncontinuous examples of $F$ (but such examples rely on the Axiom of Choice).
Best Answer
Suppose there exists $2$ fixed points, $x$ and $y$. Then $x + y > 0$, so we get $$ x^2(x + y) = (x + y)f(xy) $$ and therefore $f(xy) = x^2$. Swapping $x$ and $y$, we get $f(xy) = y^2$. Therefore $x^2 = y^2$, and $x = y$.
So, there can be at most $1$ fixed point. Now, you have shown that $1$ is a fixed point. But $xf(x)$ is a fixed point for all $x > 0$, therefore $$ f(x) = \frac{1}{x} $$ for all $x \in \mathbb R^+$. We must then check this solution. For all $x,y$ we have $$ \begin{aligned} x^2(f(x) + f(y)) &= x^2\left(\frac 1 x + \frac 1 y\right)\\ &= x + \frac{x^2}{y}\\ &= \frac{x}{y}\left(y + x\right)\\ &= (x + y)f\left(\frac{y}{x}\right)\\ &= (x + y)f(f(x)y) \end{aligned} $$ So $f\colon x \mapsto \frac{1}{x}$ is a solution, and is the only solution.
It is usually pretty hard to prove monotonicity in functional equations, without outright finding the function. To reduce the possible number of fixed points, it is however pretty common to plug in two fixed points into the equation, which makes the contibutions of $f$ "vanish".