This was posted to Math Stackexchange, but got no useful answers, and the more I think about it, the harder it seems.
I would like to know whether there exists a differentiable function from the (open or closed) unit interval to itself satisfying
$$1-x-f(f(x))-f(x)f'(f(x))=0$$
for all $x$.
Ideally, I'd also like a list of all such functions.
This arose in the course of an economics problem whose description would be off-topic here, so lest this question seem too localized, let me pose a general question:
What techniques are available for solving functional equations of the above type?
Best Answer
Write $$1-x=f(f(x))+f(x)f'(f(x)).$$ Note, that $f(x_1)=f(x_2)$ implies $x_1=x_2$ and thus the function is injective on $(0,1).$ Therefore, it is strictly monotone. This implies that $f(f(x))$ is strictly increasing.
If $f$ is increasing, then $f'(f(x))\ge 0$ and thus $1-x\ge f(f(x)).$ Letting $x\to 1$ leads to a contradiction.
If $f: (0,1)\to [0,1]$ and $f$ decreases, then $f(x)\cdot f'(f(x))\le 0$ and $1-x\le f(f(x)).$ Letting $x\to 0$ leads to a contradiction.