Question
Let $f: \mathbb R \to \mathbb R$ be a strictly increasing function with $\lim_{x \to \infty} f(x) = \infty$ and $f(x) = o(x)$.
Does it then hold that $x_n := \frac 1n \cdot f^{-1}\Big( \big[n f(n) \big]^{1/2} \Big) \to \infty$ as $n \to \infty$?
Thoughts
The idea here is to define a sequence $(x_n)$ such that $f(nx_n) = \big[ n \cdot f(n) \big]^{1/2}$, the geometric mean of $n$ and $f(n)$. (Note that this holds iff $x_n$ is defined as above.)
My idea to show this was to first show that $x_n$ is increasing. And then to show that it's unbounded by contradiction. As of yet, I haven't been successful. (Perhaps the statement's not even true.)
To show $x_n$ is increasing for large $n$, set $u(x) := \frac 1x \cdot f^{-1}\big( \sqrt{xf(x)} \big)$ and show this is an increasing function for large $x$.
If we knew that $f$ were differentiable, we could calculate the derivative
$$
\begin{aligned}
u'(x) =
\frac{\text d}{\text d x} \left[ x^{-1} \cdot f^{-1} \Big( \big[x f(x) \big]^{1/2} \Big) \right] &= x^{-1} \cdot \underbrace{ \frac{\text d}{\text d x}\left\{ f^{-1} \Big( \big[x f(x) \big]^{1/2} \Big) \right\}}_{\substack{ = [f^{-1}]'([xf(x)]^{1/2}) \cdot \frac 12 [xf(x)]^{-1/2} \cdot [f(x) + xf'(x)] }} – \frac{1}{x^2} \cdot \left\{ f^{-1} \Big( \big[x f(x) \big]^{1/2} \Big) \right\} \\
&=\frac{ [f^{-1}]'([xf(x)]^{1/2}) \cdot [f(x) + x f'(x)]}{2x [x f(x)]^{1/2}} – \frac{f^{-1} ( [x f(x)]^{1/2})}{x^2}.
\end{aligned}
$$
and I think we might somehow argue that this is positive for large $x$. That would imply that $x_n$ is increasing for large $n$.
Suppose (for the sake of contradiction) that $x_n \nrightarrow \infty$ as $n \to \infty$.
Then $x_n < K < \infty$ for some $K >0$. Then hopefully get a contradiction somehow.
Best Answer
We claim that $x_{n}$ is not bounded.
Suppose $x_{n}$ is bounded. Choose a positive integer $M > 1$ so that $x_{n} \leq M$ for all natural numbers $n$. We have
$$\forall n \in \mathbb{N} \text{ }f(Mn) \geq (nf(n))^{\frac{1}{2}}.$$
We claim by induction that for all $p \in \mathbb{N}$ ($p > 0$) we have $$f(M^{p}n) \geq M^{p+2^{1-p}-2}f(n)^{2^{-p}}n^{1-2^{-p}}$$ for all $n \in \mathbb{N}$.
Indeed for $p = 1$ this is true. Assume for $p = k$ that this is true, we have
$$f(M^{k+1}n)\geq (M^{k}n M^{k+2^{1-k}-2}f(n)^{2^{-k}}n^{1-2^{-k}})^{\frac{1}{2}} = M^{(k+1)+2^{1-(k+1)}-2}f(n)^{2^{-(k+1)}}n^{1-2^{-(k+1)}}.$$
as was required.
Thus
$$\liminf_{p \rightarrow \infty}\frac{f(M^{p}n)}{M^{p}n} \geq \frac{1}{M^2} $$
but this violates $f(x) = o(x)$ (this tells us that $\lim_{x \rightarrow \infty}\frac{f(x)}{x} = 0$).
$\textbf{Edit}:$ Your question is not true because of a counter-example. Let $\mathbb{N} = \{1,2,...\}$
Define $L$ so that
$$L(x) =\begin{cases} 2^{x-3} && x \leq 2\\ \frac{1}{2} && x =2\\ L(2^{a^2+1})+\frac{1}{a}(x-2^{a^2+1}) && a \in \mathbb{N}, 2^{a^2+1}\leq x \leq 2^{(a+1)^2} \\ L(2^{a^2})+\frac{\sqrt{2^{a^2}L(2^{a^2})}-L(2^{a^2})}{2^{a^2}}(x-2^{a^2}) && a \in \mathbb{N}, 2^{a^2}\leq x \leq 2^{a^2+1} \end{cases}$$
this function satisfies your above requirements and satisfies $L(2(2^{a^2})) = \sqrt{2^{a^2}L(2^{a^2})}$ for $a \in \mathbb{N}$ thus
$$\frac{1}{2^{a^2}}L^{-1}(\sqrt{2^{a^2}L(2^{a^2})}) = 2$$
for $a = 1,2,..$ and thus is your counterexample.