How can I find all functions $f:\mathbb{R}\to\mathbb{R}$ such that $f(1)=1$ and
$$f\left(f(x)y+\frac{x}{y}\right)=xyf\left(x^2+y^2\right)$$
for all real numbers $x$ and $y$ with $y\neq0$?
PS. This is from USA MOSP (Mathematical Olympiad Summer Program) 2007, and is one of the most difficult problems I've seen ever. This problem has been posted a couple of times on MathLinks fora, but I couldn't find a complete solution for it… And the only solution is $f(x)=\frac{1}{x}$. Please try it, thanks!
Best Answer
Here is my approach. The equation $f(x)y+\frac{x}{y}=x^2+y^2$ for $x \neq 0$ has at least one real root, since is equivalent to a cubic equation. Denote that one root with $y_0$.
Substituting $y_0$ in the functional equation we get $f(x^2+y^2)=xy_0f(x^2+y^2)$. Notice that $x^2+y_0^2\neq 0$. If we can prove that $f(c)=0$ if and only if $c=0$ we are done, since then the relation above imples $y_0=\frac{1}{x}$ and therefore, from the relation $f(x)y_0+\frac{x}{y_0}=x^2+y_0^2$ we get $f(x)=\frac{1}{x}$.
Therefore, to prove that the only solution of the functional equation is the one mentioned above, we just need to prove the following:
I've made some small steps in proving this but the proof is not complete.
The relation $f(f(x)+x)=xf(x^2+1)$ allows us to prove that if $x \neq 0$ then $f(x)=0\Rightarrow f(x^2+1)=0$. This implies that there exists a sequence $K_n \to \infty$ such that $f(K_n)=0$.
Assume that $f$ is continuous.
Take now $x,y$ with $x^2+y^2=K_n$. The initial relation implies that $f(f(x)y+\frac{x}{y})=0,\ \forall x^2+y^2=K_n$. For $y \to 0+$ and $x>0$ we get that $f(x)y+\frac{x}{y} \to \infty$ (since $x \to \sqrt{K_n})$ This means that $f$ takes zero values in a neighborhood of $\infty$. Since $f(y+\frac{1}{y})=yf(1+y^2)$ implies $f(x)=-f(-x),\forall x$ with $|x| \geq 2$, we have the same thing in a neighborhood of $-\infty$.
Notice that for $y >1 $ we have $y^2+1>y+\frac{1}{y}$, and we can translate the "zeros" in the neighborhood of $\infty$ until we reach $2$, i.e. $f(x)=0$ on $(-\infty,-2]\cup [2,\infty)$.
I don't know where to go from here. This is not an answer, and I added an extra hypothesis towards the end, but I think this might lead to a result.
Maybe there are some more general solutions such as
$$ f(x)=\begin{cases} \frac{1}{x}, &x \in A \\ 0, & x \notin A \end{cases}$$, where $A$ is a set with some given properties.