EDİT: I think I've repair the error in the solution. I want to know if I'm fixing it properly. I'm just a student, not a mathematician. Please focus on the "backbone" of my writing.
I want to prove that, the following substitution is correct:
$$ x\mapsto f^{-1}(x)$$
Statement: For a function to have an inverse, each element $y ∈ Y $ must correspond to no more than one $x ∈ X; $a function $f$ with this property is called one-to-one or an injection.
The function $f(x)$ is an injective and $f$ is an invertible function. In other words, if $f(0)≠0$, for function $f(x)$, the inverse function $f^{-1}(x)$ is exist.
$f(x)=f(0)-\frac {x}{f(0)} \Rightarrow f^{-1}(x)=f(0)(f(0)-x)$ and $f(f^{-1}(x))=x$ for all $x\in\mathbb {R}$. For this reason, we can applying the substitution $ x\mapsto f^{-1}(x), x\in\mathbb{R}.$
For example:
If $f(f(x))=f(x)-5 , x\in\mathbb{R}$, then $f(x)=x-5$ must be. Because, the function $f(x)$ is an injective and $f$ is an invertible function. Applying $x\mapsto f^{-1}(x)$ we have $f(x-5)=x-10 \Rightarrow f(x)=x-5.$
I hope you can understand what I mean. Can you tell me that this approach is wrong or true?
Problem: Let $\mathbb{R}$ be the set of real numbers. Determine all functions $f: \mathbb{R} \to\mathbb{R}$ such that, for all real numbers $x$ and $y$, $$f \big(f(x)f(y)\big) + f(x +y) = f(xy).$$
I ask You to confirm that the solution is sufficient / insufficient / missing / incorrect or correct. Here is my attempts:
Are there any problems in my substitutions?
Best Answer
Empy2 pointed out the error in your proof. You cannot deduce that $f\big(f(x)f(0)\big)+f(x)=f(0)$ for all $x$ implies $f\big(xf(0)\big)+x=f(0)$ for all $x$. This only works only for $x$ in the range of $f$. I am presenting a different solution.
If $f(0)=0$, as you showed, we have $$f(0)+f(x)=f\big(f(x)f(0)\big)+f(x+0)=f(x\cdot 0)=f(0)$$ so $f(x)=0$ for all $x$. We assume that $f(0)\ne 0$ from now on. We claim that $f(c)=0$ implies that $c=1$.
Suppose that $f(c)=0$ for some $c\ne 1$. Then, we have $$f\Biggl(f(c)f\left(\frac{c}{c-1}\right)\Biggr)+f\left(c+\frac{c}{c-1}\right)=f\Biggl(c\left(\frac{c}{c-1}\right)\Biggr).$$ That is, $$f(0)+f\left(\frac{c^2}{c-1}\right)=f\left(\frac{c^2}{c-1}\right),$$ or $f(0)=0$, contradicting the assumption that $f(0)\ne 0$. Therefore, the hypothesis that $c\ne1$ is false.
Now, we note that for every $x\ne1$, $$f\Biggl(f(x)f\left(\frac{x}{x-1}\right)\Biggr)+f\left(x+\frac{x}{x-1}\right)=f\Biggl(x\left(\frac{x}{x-1}\right)\Biggr),$$ so $$f\Biggl(f(x)f\left(\frac{x}{x-1}\right)\Biggr)+f\left(\frac{x^2}{x-1}\right)=f\left(\frac{x^2}{x-1}\right).$$ So, $$f\Biggl(f(x)f\left(\frac{x}{x-1}\right)\Biggr)=0$$ and we then conclude from our claim above that $$f\left(\frac{x}{x-1}\right)=\frac{1}{f(x)}\tag{1}$$ for every $x\ne 1$. In particular, this shows that $f(1)=0$, since there does exist $c$ such that $f(c)=0$.
From (1), we get $f(0)=\pm 1$. Observe that $f$ is a solution if and only if $-f$ is also a solution. We can without loss of generality assume that $f(0)=-1$. Plugging $y\mapsto 1$ in the original functional equation, we have $$-1+f(x+1)=f(0)+f(x+1)=f\big(f(x)f(1)\big)+f(x+1)=f(x\cdot1)=f(x),$$ or $$f(x+1)=f(x)+1.\tag{2}$$ By induction, $f(x+n)=f(x)+n$ for all integers $n$.
Here, we claim that $f$ is injective. Suppose that $f(u)=f(v)$ for some $u,v\in\mathbb{R}$. Pick a positive integer $n$ so large that $4(u+n)< (v+n+1)^2$. Hence, there are two distinct $a,b\in\mathbb{R}$ such that $u+n=ab$ and $v+n+1=a+b$ (since $a$ and $b$ are the roots of the polynomial $t^2-(v+n+1)t+(u+n)$, which have two distinct real roots). Therefore, $$f\big(f(a)f(b)\big)+f(a+b)=f(ab)=f(u+n)=f(u)+n.$$ But $f(a+b)=f(v+n+1)=f(v)+n+1=f(u)+n+1$. That is, by (2), we have $$f\big(f(a)f(b)+1\big)=f\big(f(a)f(b)\big)+1=0.$$ This means $f(a)f(b)+1=1$, or $f(a)f(b)=0$. Consequently, $f(a)=0$ or $f(b)=0$, which means $a=1$ or $b=1$. Without loss of generality, $b=1$, so $u+n=ab=a$ and $v+n+1=a+b=a+1$. That is, $u=a-n=v$.
Now, substitute $y\mapsto 1-x$ in the original functional equation. We then have $$f\big(f(x)f(1-x)\big)=f\big(f(x)f(1-x)\big)+f(1)=f\big(x(1-x)\big).$$ Thus, by injectivity, $$f(x)f(1-x)=x(1-x).$$ Then, we take $y\mapsto -x$ in the original functional equation to get $$f\big(f(x)f(-x)\big)-1=f\big(f(x)f(-x)\big)+f(0)=f\big(x(-x)\big).$$ Thus, $$f\big(f(x)f(-x)\big)=f(-x^2)+1=f(-x^2+1)$$ by (2). By injectivity, $$f(x)f(-x)=-x^2+1.$$ From (2), we also have \begin{align}f(x)&=f(x)\Big(\big(f(-x)+1\big)-f(-x)\Big)\\&=f(x)\big(f(1-x)-f(-x)\big)=f(x)f(1-x)-f(x)f(-x),\end{align} so $$f(x)=x(1-x)-(-x^2+1)=x-1.$$
It is easy to see that $f(x)=x-1$ is indeed a solution. Therefore, all solutions are $f(x)=0$, $f(x)=x-1$, and $f(x)=1-x$.