There is no such $f$.
From http://yaroslavvb.com/papers/rice-when.pdf , the question of existence is determined by the theorem:
Theorem 6. Let $\mathbb{R}$ be the real line. Let $g$ be a real quadratic polynomial, so that
$$g(x)=ax^2+ (b + 1)x+c,$$
for all real $x$, where $a\ne 0$, $b$, and $c$ are in $\mathbb{R}$. ... set $\Delta(g)= b^2-4ac$. If $\Delta(g)> 1$,
then g has no iterative roots of any order whatever. [That is, there is no $f$ such $f\circ f = g$.] If $\Delta(g) =1$, then $g$ can be embedded in a 2-sided
flow on $\mathbb{R}$, all of whose members are continuous functions. If $\Delta(g) <1$, then $g$ can be embedded in a
1-sided flow on $\mathbb{R}$, all of whose members are continuous functions; but $g$ cannot be embedded in any
2-sided flow on $\mathbb{R}$.
As $\Delta(g) = 0 - 4(1)(-2) = 8 > 1$ in your case, the question of existence is negative.
Looking closely at the article, the main point is that no function with only one 2-cycle can have a square root. In our case that means that there can be no partial solution $f:D\to D$ of the funcional equation $f(f(x))=x^2-2$ in $D\subset\Bbb{R}$ if $x_0=\frac{-1+\sqrt{5}}{2}\approx 0.61803$ or
$x_2=\frac{-1-\sqrt{5}}{2}\approx -1.61803$ are in $D$.
In fact, clearly $x_0^2-2=x_2$ and $x_2^2-2=x_0$ (this implies that $x_0\in D$ if and only if $x_2\in D$).
There can be no other pair $y_1\ne y_2$ with $y_1^2-2=y_2$ and $y_2^2-2=y_1$, since then
$$
\{-1,2,x_0,x_2,y_1,y_2\}
$$
would be roots of the polynomial $P(x)=x^4 - 4 x^2 - x + 2$, since
$y_1^2-2=y_2$ and $y_2^2-2=y_1$ implies
$$
(y_1^2-2)^2-2=y_1\quad\Rightarrow \quad y_1^4-4y_1^2+2=y_1
\quad\Rightarrow \quad P(y_1)=0
$$
and similarly $P(y_2)=0$.
Now, if $x_0\in D$ (or $x_2\in D$) and $f:D\to D$ satisfy $f(f(x))=x^2-2$, then $x_1:=f(x_0)$ and $x_3:=f(x_2)$ would be such a pair, a contradiction that proves $x_0\notin D$ (and $x_2\notin D$).
This has the air of exploiting a hole left in the question parameters, but here comes.
Let
$D=\{0\}\cup(1,\infty)$ and let $f(x)$ be a constant function. Then $T$ will be a period if and only if $T+D\subseteq D$. In particular:
- every $T>1$ is a period, but
- $T=1$ is not a period (and there cannot be smaller periods $\le 1$),
- so there is no smallest period.
Best Answer
An important piece of information is:
Theorem: $f$ is not continuous.
Proof: Observe that $f$ is invertible, because
$$f(f(f(f(x)))) = f(f(-x)) = x$$
and so $f \circ f \circ f = f^{-1}$. Any continuous invertible function on $\mathbb{R}$ is either strictly increasing or strictly decreasing.
If $f$ is strictly increasing, then:
contradiction! Similarly, if $f$ is strictly decreasing, then:
contradiction! Therefore, we conclude $f$ is not continuous. $\square$
For the sake of completeness, the entire solution space for $f$ consists of functions defined as follows:
To see that every solution is of this form, let $f$ be a solution. Then we must have $f(0) = 0$ because:
If $a \neq 0$, then let $f(a) = b$. We have:
From here it's easy to see the set $\{ (a,f(a)) \mid a>0, f(a)>0 \}$ partitions the positive real numbers and so is of the form I describe above.
One particular solution is
$$ f(x) = \begin{cases} 0 & x = 0 \\ x+1 & x > 0 \wedge \lceil x \rceil \text{ is odd} \\ 1-x & x > 0 \wedge \lceil x \rceil \text{ is even} \\ x-1 & x < 0 \wedge \lfloor x \rfloor \text{ is odd} \\ -1-x & x < 0 \wedge \lfloor x \rfloor \text{ is even} \end{cases}$$
e.g. $f(1/2) = 3/2$, $f(3/2) = -1/2$, $f(-1/2) = -3/2$, and $f(-3/2) = 1/2$.
(This works out to be Jyrki Lahtonen's example)