First, $f= 0$ is a solution, as pointed out by Travis. Now, assume $f\neq 0$.
Let $d:=\deg(f)$, let $a_d$ be the leading coefficient. Divide both sides of the functional equation by $x^{2d}$ and let $x\to \infty$. It follows that $a_d^2=a_d$ and $a_d\neq 0$, so $a_d=1$. In other words, $f$ is monic. Now, let $f(x)=:x^d+g(x)$, then the functional equation becomes
$$
\begin{align*}
(g(x)+x^d)(g(x+1)+(x+1)^d)&=g(x^2+x)+(x^2+x)^d\\
g(x)g(x+1) + x^dg(x+1)+(x+1)^dg(x)=g(x^2+x).
\end{align*}
$$
If $g\neq 0$, the degree on the left is $d+\deg(g)$. This is strictly larger than $2\deg g$, which is the degree on the right. We conclude that $g=0$ and $f=x^d$. Indeed, this polynomial satisfies the functional equation.
Claim: all continuous solutions $f:(1,\infty)\to\mathbb R$ take the following form: $$f(x) = g(\log_2 \log_2 x) + \log_2\log_2 x$$ where $g:\mathbb R\to\mathbb R$ is any continuous function of period $1$. Further, the above defines a bijection between the set of continuous functions $(1,\infty)\to \mathbb R$ satisfying the given functional equation, and the set of continuous functions with period $1$. (So you can think of the periodic continuous functions as "parametrizing" solutions to your functional equation.)
Proof. Let $\mathcal C^0(\mathbb T^1)$ denote the set of continuous real-valued functions with period one (i.e. "functions on the 1-torus") and let $F\subset \mathcal C^0(\mathbb R)$ denote the set of continuous functions solving the given functional equation. We will show that the above defines a function $\Phi:\mathcal C^0(\mathbb T^1)\to F$, and moreover that this function is bijective.
First, let $g\in \mathcal C^0(\mathbb T^1)$ be an arbitrary continuous periodic real-valued function. We know that the logarithm base $2$ is defined and continuous on $(0,\infty)$, and that it maps $(1,\infty)$ onto $(0,\infty)$ bijectively, hence $\log_2\log_2 x$ defines a continuous function on $(1,\infty)$. This means that $\Phi(g)$ as defined earlier is a continuous real-valued function $(1,\infty)\to\mathbb R$. Further, we can easily show that it satisfies the required functional equation:
\begin{align}
f(x^2)
&= g(\log_2 \log_2 x^2) + \log_2 \log_2 x^2 \\
&= g(\log_2(2\log_2 x)) + \log_2(2\log_2 x) \\
&= g(\log_2\log_2 x + 1) + \log_2\log_2 x + 1 \\
&= g(\log_2\log_2 x) + \log_2\log_2 x + 1 \\
&= f(x) + 1 \\
\end{align}
This shows that $\Phi:\mathcal C^0(\mathbb T)\to F$ is well-defined.
To see that it is injective, suppose that $\Phi(g_1) = \Phi(g_2)$. Then, for any $x\in\mathbb R$, we have that $\Phi(g_1)(2^{2^x}) = \Phi(g_2)(2^{2^x})$, since $2^{2^x} > 1$ for all $x\in\mathbb R$, and therefore $2^{2^x}$ is in fact an element of the domains of these two functions. But substituting the definition of $\Phi$ into this equation yields $$g_1(x) + x = g_2(x) + x$$ and therefore $g_1(x) = g_2(x)$. Since $x$ was arbitrary, we conclude $g_1 = g_2$. Thus $\Phi$ is injective.
Finally we show that $\Phi$ is surjective. Let $f$ be an arbitrary continuous solution to your functional equation, and define $$g(x) = f(2^{2^x}) - x$$ Clearly $g$ is well-defined because $2^{2^x} > 1$ for all real $x$, and it is continuous because $2^{2^x}$ is a continuous function of $x$, and $f$ is also continuous. Further, we can see that $g$ has period $1$, because
\begin{align}
g(x+1)
&= f(2^{2^{x+1}}) - x - 1 \\
&= f((2^{2^{x}})^2) - x - 1 \\
&= f(2^{2^{x}})+1 - x - 1 \\
&= f(2^{2^{x}}) - x \\
&= g(x)
\end{align}
so we may conclude that $\Phi(g) = f$, hence $\Phi$ surjective. This completes our proof.
Best Answer
We can write that $$ x = f(f^{-1}(x)) = f(x - f(x)). $$
Notice that the left hand side can assume any value on $\mathbb{R}$, so the image of $f$ is $\mathbb{R}$.
Also, since $f$ is invertible it must be injective.
In particular, there is a point $x_0$ such that $f(x_0) = 0$. Substituting on the previous equation, we conclude that $x_0 = f(x_0 - 0) = 0$. We have then $f(0) = 0$.
We claim that $0$ is the only fixed point of $f$. Indeed, let $x_1$ be some fixed point of the function. Then $x_1 = f(f(x_1) - x_1) = f(0) = 0$.
We have $f$ is a monotone function, as a consequence of the intermediate value theorem.
We claim that $f$ cannot be increasing. Indeed, assume that $f$ is increasing. Then, $f^{-1}$ is also increasing. Then, for $x > 0$ since $f(x) \neq x$ we have two possible cases:
We claim that $f$ cannot be decreasing. If it were, then for $x > 0$ we would have $f(x) < 0 < x$ and applying the also decreasing function $f^{-1}$, one would get $x > 0 > f^{-1}(x)$. Thus, we conclude that $x = f(x) + f^{-1}(x) < 0 < x$, contradiction.
Since $f$ cannot be decreasing nor increasing, we conclude that there is no $f$ satisfying the equation.