I wish to find all continuous functions $f: \mathbb{R} \to \mathbb{R}$ that satisfy $$f\big(f(x)\big)=rf(x)+sx\quad\forall x\in\mathbb{R}\,,$$ where $r,s \in (0, 1/2)$.
Here's my work so far:
Let $r_1 > r_2$ be the roots of $x^2-rx-s.$ We have $1>|r_1|>|r_2|$ and $r_2<0<r_1.$ Furthermore, $f(x)=r_1 x, f(x)=r_2 x$ are clearly solutions to the FE. If $f(x)=f(y),$ then $rf(y)+sy = f(f(y))=f(f(x))=rf(x)+sx \Rightarrow x=y.$ Thus, $f$ is injective, which means $f$ is either strictly increasing or strictly decreasing. If $f$ is bounded below or above, then $sx = f(f(x))-rf(x)$ is as well, which is absurd. Thus, $f(\mathbb{R})=\mathbb{R}$ and $f$ is surjective. Now assume $f$ is increasing. I suspect that we need to show $f(x)=r_1 x$ from here, and the strictly decreasing case will be similar.
Let $a_0 = x_0, a_{n+1} = f(a_n).$ Then $a_{n+2}-ra_{n+1}-sa_n = 0,$ This is a linear recursion with characteristic polynomial $x^2-rx-s,$ so $a_n = c_1 r_1^n + c_2 r_2^n$ for some $c_1, c_2.$ Solving for the constants, we get $c_1 = \frac{f(x_0) – x_0 r_2}{r_1-r_2}, c_2 = \frac{x_0 r_1 – f(x_0)}{r_1-r_2}.$ I managed to find $f(0) = 0$ by taking $n \to \infty,$ but did not get any use out of this observation.
Another idea that I remembered could come in helpful is the fact that $x>y \Rightarrow f(x)>f(y)$ for increasing functions. Perhaps if we assume $f(x_0) > rx_0$ for some $x_0,$ we could derive a contradiction. $f(x_0) < rx_0$ would lead to a similar contradiction, and we would be done. Using $f$ on both sides $n$ times, we get $f^{n+1}(x_0) > f^{n}(rx_0),$ which simplifies to $f(x_0) > f(r_1 x_0) \cdot \frac{r_1^n – r_2^n}{r_1^{n+1}-r_2^{n+1}} + r_2 r_1^n (r_2-r_1)x_0$ after some algebraic manipulation. If we use the fact that $|r_2/r_1| < 1$ and take $n \to \infty,$ we get $f(x_0) > f(r_1 x_0)/r_1 \Rightarrow r_1f(x_0) > f(r_1 x_0).$ Unfortunately, this by itself is not a contradiction. You can compose both sides with $f$ again and take the limit once more, but that seems like it won't lead anywhere.
Update: Composing both sides with $f$ and taking the limit once more gives $f(r_1 f(x_0)) > r_1(r_2f(x_0)+f(r_1 x)+r_1r_2x_0).$ Still no contradiction.
After all this work, I have exhausted my bag of tricks. What else can I use on this problem? What would be the motivation behind these additional approaches?
Best Answer
In this solution, for a nonnegative integer $n$, $f^{\circ n}$ denotes the $n$-time iteration of $f$ (with $f^{\circ 0}$ being the identity function), and for a negative integer $n$, $f^{\circ n}$ is the functional inverse of $f^{\circ |n|}$. I quote your results $f(0)=0$ and, for every $n\in\mathbb{Z}$ (not just $n\in \mathbb{Z}_{\geq 0}$), $$f^{\circ n}(x)=\frac{f(x)-r_2x}{r_1-r_2}\,r_1^n+\frac{r_1x-f(x)}{r_1-r_2}\,r_2^n\,,$$ where $r_1>0>r_2$ with $1>|r_1|>|r_2|$. I shall prove that $f(x)=r_1x$ for all $x\in\mathbb{R}$ if $f$ is increasing, and $f(x)=r_2x$ for all $x\in\mathbb{R}$ if $x$ is decreasing.
If $f$ is increasing, then $f^{\circ n}(x)>0$ for all $x>0$ and $n\in\mathbb{Z}$. In particular, when $n$ is a large negative integer, we can see that the term $r_2^n$ dominates the term $r_1^n$ in $f^{\circ n}(x)$. Thus, unless $f(x)=r_1x$, the $r_2^n$-term in $f^{\circ n}(x)$ will oscillate between positive and negative values, contradicting with $f^{\circ n}(x)>0$. Thus, $f(x)=r_1x$ for $x>0$. A similar argument shows that $f(x)=r_1x$ for $x<0$. Ergo, $f(x)=r_1x$ for all $x\in\mathbb{R}$.
If $f$ is decreasing, then $(-1)^n\,f^{\circ n}(x)>0$ for all $x>0$ and $n\in\mathbb{Z}$. In particular, when $n$ is a large positive integer, we can see that the term $(-r_1)^n$ dominates the term $(-r_2)^n$ in $(-1)^n\,f^{\circ n}(x)$. Thus, unless $f(x)=r_2x$, the $(-r_1)^n$-term in $(-1)^n\,f^{\circ n}(x)$ will oscillate between positive and negative values, contradicting with $(-1)^n\,f^{\circ n}(x)>0$. Thus, $f(x)=r_2x$ for $x>0$. A similar argument shows that $f(x)=r_2x$ for $x<0$. Ergo, $f(x)=r_2x$ for all $x\in\mathbb{R}$.