I would like to give a solution of both the instances of the problem which appeared here in different times, i.e. the version posted before the OP edited his post and the current one. First I want to solve the problem as it is stated by the OP.
Problem I
$$f(y+f(x))=f(x)f(y)+f(f(x))+f(y)-xy.$$
As already noticed, setting $y=0$, one arrives to $$f(f(x))=f(0)f(x)+f(f(x))+f(0)$$ which leads to $f(x)\equiv -1$, which is not a solution of the problem, or $f(0)=0$. Incidentally, even though it is not crucial for the solution, notice that if we assume that there is an $\bar x$ such that $f(\bar x)=0$, setting $x=\bar x$ we infer that $f(y)=f(y)-\bar xy$. In particular, setting $y=1$, we conclude $\bar x=0$.
Next, plug inthe equation $f(y)$ instead of $y$ to get $$f(f(y)+f(x))=f(x)f(f(y))+f(f(x))+f(f(y))-xf(y).$$ Interchange the roles of $x$ and $y$ to obtain similarly $$f(f(x)+f(y))=f(f(x))f(y)+f(f(y))+f(f(x))-yf(x).$$ Compare these two equations to conclude that $$\frac{f(f(x))+x}{f(x)}=\frac{f(f(y))+y}{f(y)}=k,$$ where $k$ is a suitable constant.
Substitute in the original equation to get $$f(y+f(x))=f(x)f(y)-x+kf(x)+f(y)-xy.$$ Denote $a:=f(-1)$ and choose $y=-1$ to obtain $$f(f(x)-1)=(a+k)f(x)+a$$
Edit Thanks to Dejan who pointed out a flaw in my proof
Plug in the original equation $f(y)-1$ instead of $y$ to obtain $$\begin{aligned}f(f(y)+f(x)-1)&=f(x)[(a+k)f(y)+a]-x+kf(x)-x(f(y)-1)\\&=(a+k)(f(x)+f(y))-xf(y).\end{aligned}$$ Exchange $x$ and $y$ to obtain $$f(f(y)+f(x)-1)=(a+k)(f(x)+f(y))-yf(x).$$ Compare these two equations to deduce that $$\frac{f(x)}{x}=\frac{f(y)}{y}=a.$$
Substitute in the original equation to find out that the admissible values for $a$ are just $\pm 1$ i.e. $$f(x)=\pm x.$$
Problem II
Now let's see what happens if the equation is written instead as $$f(x+f(y))=f(x)f(y)+f(f(x))+f(y)-xy.$$ Setting $x=y=0$ one deduces that $$f(0)^2+f(0)=0$$ from which $f(0)\in \{-1, 0\}$.
Assume then $f(0)=0.$ Then plug in the equation $x=0$ to deduce that $f(y)=f(f(y)).$
Let's see that $f$ is injective. Suppose there are $y\neq z$ with $f(y)=f(z)$. Use the equation with $y$ and $z$ and deduce that in this case $x(y-z)=0$ should hold for all $x\in\mathbb R$, which is plainly impossible. Then $f$ is injective. Now, recall that in our hypothesis the equation states as $$f(x+f(y))=f(x)f(y)+f(x)+f(y)-xy.$$ Exchange $x$ and $y$ to obtain $$f(y+f(x))=f(y)f(x)+f(y)+f(x)-yx.$$ It follows from the injectivity of $f$ that $$f(x)-x=f(y)-y=k=0.$$ It is easy to verify that the identity is indeed a solution of our equation.
Assume now $f(0)=-1$. Then, choosing $x=0$, one deduces $$f(f(y))=f(f(-1))=a.$$ Substitute then $x$ with $f(x)$ to derive $$f(f(x)+f(y))=af(y)+f(a)+f(y)-f(x)y.$$ Similarly, also $$f(f(x)+f(y))=af(x)+f(a)+f(x)-f(y)x$$ holds. Therefore we finally find that $$\frac{f(x)}{(a+1+x)}=\frac{f(y)}{(a+1+y)}=c.$$ It follows that there are constants $c,d\in\mathbb R$ such that $$f(x)=cx+d.$$ From the fact that $f(f(x))=a$, we deduce $c=0$, and from $f(0)=-1$, it follows $f\equiv -1$ which is not a solution of the problem.
Consider any function $g : [1,+\infty) \rightarrow \mathbb{R}$ such that $g(1)=2$.
Then the function $f : \mathbb{R}_+^* \rightarrow \mathbb{R}$ defined by
$$f(x)=g(1/x) \quad \text{if } 0<x<1 \quad \quad \quad \text{and} \quad \quad f(x)=g(x) \quad\text{if } x\geq 1 $$
is a solution to your problem. (So there are many solutions)
(If you want continuous solutions, just ask that $g$ is continuous, it will give a continuous $f$. If you want differentiable solutions, then try to see at which condition on $g$ the constructed function will be differentiable - hint : you will have a condition about the derivative of $g$ at $1$)
Best Answer
First, observe that the integral inherits all the symmetries of the cosine function: $$I(a)=I(a+2\pi)=I(2\pi-a)$$ Invoking continuity as well, it suffices to consider $a\in (0,\pi)$.
Now consider that $$I'(a)=\int_0^1 \frac{2\sin a~dx}{x^2-2x\cos a+1}=\int_0^1 \frac{2\sin a~dx}{(x-\cos a)^2+\sin^2 a}.$$ Under the substitution $x=\cos a+u \sin a$, the corresponding indefinite integral yields
\begin{align} \int \frac{2\sin a~dx}{(x-\cos a)^2+\sin^2 a} &= \int \frac{2\sin a~ (\sin a~du)}{(1+u^2)\sin^2 a}\\ &=2\int \frac{du}{1+u^2}\\ &=\tan^{-1}u=\tan^{-1}\left(\frac{x-\cos a}{\sin a}\right) \end{align} Note that this substitution is well-defined for all $a\in (0,\pi)$. Therefore the definite integral evaluates to \begin{align} I'(a) &=2\tan^{-1}\left(\frac{1-\cos a}{\sin a}\right)-2\tan^{-1}\left(\frac{0-\cos a}{\sin a}\right)\\ &=2\tan^{-1}\left(\frac{2\sin^2 (a/2)}{2\sin (a/2)\cos(a/2)}\right)+ 2\tan^{-1}\cot(a)\\ &=2\tan^{-1}\tan\left(\frac{a}{2}\right)+2\tan^{-1}\tan\left(\frac{\pi}{2}-a\right)\\ &=2\cdot \frac{a}{2}+2\cdot\left(\frac{\pi}{2}-a\right)=\pi-a \end{align} where in the second line we have used the double-angle trig identities; the simplifications of inverse tangent are valid so long as $0<a/2<\pi$ and $0<\pi/2-a<0$, both of which follow from $a\in(0,\pi)$. Integrating this result yields $$I(a)=-\frac{1}{2}(\pi-a)^2+C$$ It remains to evaluate $I(a)$ for some value of $a$ to determine the value of $C$. To relate this to a familiar result, my preference is to evaluate in the limit $a\to 0^+$ by expanding term-by-term: \begin{align} \lim_{a\to 0^+}I(a)=C-\frac{\pi^2}{2} &=\int_0^1 \log(1-2x+x^2)\frac{dx}{x}\\ &=2\int_0^1 \log(1-x)\frac{dx}{x}\\ &=-2 \int_0^1 \sum_{k=1}^\infty \frac{x^{k-1}}{k}dx\\ &=-2 \sum_{k=1}^\infty \frac{1}{k^2}=-2\frac{\pi^2}{6}=-\frac{\pi^2}{3} \end{align} Hence $C=I(\pi)=\pi^2/6$ and we conclude $$I(a)=\frac{\pi^2}{6}-\frac{(\pi-a)^2}{2},\qquad 0<a<\pi$$ Invoking the symmetries and continuity of the original integral, this extends to $$I(a)=\frac{\pi^2}{6}-\frac12 [\pi-(a\mod 2\pi)]^2$$ which is valid for for all real $a$.