A function $f:\mathbb{R} \to \mathbb{R}$ is called real-analytic if for each $x_0 \in \mathbb{R}$ there exists a neighbourhood of $x_0$ where $f$ is given by a convergent power series centred at $x_0$.
Problem: Is there a real-analytic monotone function $f:(0,\infty) \to \mathbb{R}$ which vanishes at infinity, but whose derivative admits no limit as $x \to \infty$?
We can note some weaker, but related, results. The (non-monotone) function $f(x)=x^{-1} \sin x^2$ is a real-analytic function on $(0, +\infty)$ and has the property that $\lim_{x \to +\infty} f(x) = 0$ but $\lim_{x \to + \infty} f'(x)$ fails to exist. It's not difficult to construct monotone examples if real-analyticity is weakened to merely being infinitely differentiable. The basic construction is straightforward. For each integer $n \geq 2$, and on each interval $[n, n+1-1/n^3]$, set $f(x)=1/n$, and on intervals $[ n+1-1/n^3, n+1]$ the function is linear, and decreasing from $\frac{1}{n}$ to $\frac{1}{n+1}$. This function is piecewise linear, and not smooth at the transition points, but it's trivial to smoothen this construction by utilizing appropriate variants of $\exp(1/x)$, rather than a linear interpolation. By the mean value theorem, we have that $\sup_{x \in [n+1-1/n^3, n+1]} |f'(x)| \geq \left|\frac{\frac{1}{n+1} – \frac{1}{n}}{\frac{1}{n^3}}\right|=\frac{n^3}{n(n+1)} \xrightarrow{n \to + \infty} + \infty$ hence $\lim f'(x)$ fails to exist.
However, I don't think one can use these ideas to obtain a real-analytic monotone function with the desired properties, since there's no real-analytic "transition" functions.
Best Answer
What we need is a real-analytic non-negative and integrable $g$ that has no limit at $+\infty$. Then $$f(x) = \int_x^{+\infty} g(t)\,dt$$ fits the bill.
Consider $$g(x) = \biggl(\frac{2 + \cos x}{3}\biggr)^{6 x^5}\,.$$
It is evident that $g$ is strictly positive, real-analytic on $(0,+\infty)$, and has no limit as $x \to +\infty$. It remains to see that $g$ is integrable. For a positive integer $n$, consider the interval of length $\pi$ with midpoint $n\pi$. In this interval, for $\lvert x - n\pi\rvert \geqslant \frac{1}{n^2}$ we have $$\lvert \cos x\rvert \leqslant \cos \bigl(n^{-2}\bigr) \leqslant 1 - \frac{1}{2n^4} + \frac{1}{24n^8} \leqslant 1 - \frac{1}{3n^4}$$ by Taylor expansion, and hence (using $\bigl(n - \frac{1}{2}\bigr)\pi > \frac{3}{2}n$) $$g(x) \leqslant \biggl(1 - \frac{1}{9n^4}\biggr)^{9n^5} \leqslant \exp \bigl(-n\bigr)\,.$$ Hence the integral of $g$ over that interval is bounded by $$\frac{2}{n^2} + \pi\cdot e^{-n}\,,$$ which is a summable sequence.