I have spent some time using gp-pari. There is, of course, a formal power series solution to
$ f(f(x)) = \sin x.$ It is displayed below, identified by the symbol $g$ because I am not entirely sure whether it is a function of anything.
On the other hand, should the coefficients continue to (by and large) decrease, this suggests a nonzero radius of convergence. If the radius of convergence is nonzero, then inside that, not only is a function defined and, you know, analytic, but the functional equation is satisfied. Indeed, all that is necessary is radius of convergence strictly larger than $\frac{\pi}{2}$ owing to certain symmetries. For instance, given my polynomial $g,$ it seems we have $g=1$ at about $x \approx 1.14.$ Then we seem to have a local maximum at $x =\frac{\pi}{2},$ and apparently there $g \approx 1.14,$ strictly larger than 1 which is an important point. So everything would fall into place with large enough nonzero radius of convergence.
$$
\begin{array}{lll}
g & = & x – \frac{x^3 }{ 12} – \frac{x^5 }{ 160} – \frac{53 x^7 }{ 40320} – \frac{23 x^9 }{71680} – \frac{92713 x^{11}}{1277337600} – \\\
& & \\\
& & \frac{742031 x^{13} }{79705866240} + \frac{594673187 x^{15} }{167382319104000} + \frac{329366540401 x^{17} }{91055981592576000} + \\\
& & \\\
& & \frac{104491760828591 x^{19} }{62282291409321984000} + \frac{1508486324285153 x^{21} }{4024394214140805120000} + \cdots
\end{array}
$$
Note that the polynomial $g$ is smaller than $x$ but larger that $\sin x,$ for, say,
$0 < x \leq \frac{\pi}{2}.$
So, that is the question, does the formal power series beginning with $g$ converge anywhere
other than $x = 0$?
EDIT: note that the terms after the initial $x$ itself have all turned out to be
$$ \frac{a_{2 k + 3} x^{2 k + 3} }{2^k ( 2 k + 4)!} $$
where each $a_{2 k + 3}$ is an integer. This much seems provable, although I have not tried yet.
EDIT, Friday 12 November 2010. It now seems really unlikely that this particular problem gives an analytic answer. I suspect that the answer is $C^\infty$ and piecewise analytic, with failure of analyticity at only the points "parabolic" where the derivative has absolute value as large as 1, those points being $0,\pi, 2 \pi, \ldots.$ However, we need the anchor point at the fixpoint 0, otherwise how to begin? And I do think the power series will serve as an asymptotic expansion around 0.
Given the problem with the size of the derivative, now I am hoping for great things, and an obviously periodic and analytic solution, to the easier variant $f(f(x)) = g(x) = (1/2) \sin x.$ I would like both a nice power series and a nice answer by methods summing iterates $ g^{[k]}(x),$ which for the moment is an entirely mysterious method to me, but attractive for periodic target functions as periodicity would be automatic.
Best Answer
EDIT, September 2014: I wrote to Prof. Ecalle, it turns out (as I had hoped) that the fractional iterates constructed by the recipe below really do come out $C^\infty,$ including a growth bound, in terms of $n,$ on the $n$-th derivatives at $0.$ The key word phrase is Gevrey Class. Also, I recently put a better exposition and example of the technique at https://math.stackexchange.com/questions/911818/how-to-get-fx-if-we-know-ffx-x2x/912324#912324
EDIT Feb. 2016: given that there is new discussion of this, i am pasting in the mathematical portion of Prof. Ecalle's reply, which includes the references
The "Six Lectures" are in Schlomiuk editor, 1993, Bifurcations and periodic orbits of vector fields / edited by Dana Schlomiuk. The reference is currently number 19 on Ecalle's web page, it reads:
ORIGINAL: The correct answer to this belongs to the peculiar world of complex dynamics. See John Milnor, Dynamics in One Complex Variable.
First, an example. Begin with $f(z) = \frac{z}{1 + z},$ which has derivative $1$ at $z=0$ but, along the positive real axis, is slightly less than $x$ when $x > 0.$ We want to find a Fatou coordinate, which Milnor (page 107) denotes $\alpha,$ that is infinite at $0$ and otherwise solves what is usually called the Abel functional equation, $$ \alpha(f(z)) = \alpha(z) + 1.$$ There is only one holomorphic Fatou coordinate up to an additive constant. We take $$ \alpha(z)= \frac{1}{ z}.$$ To get fractional iterates $f_s(z)$ of $f(z),$ with real $0 \leq s \leq 1,$ we take $$ f_s (z) = \alpha^{-1} \left( s + \alpha(z) \right) $$ and finally $$f_s(z) = \frac{z}{1 + s z}.$$ The desired semigroup homomorphism holds, $$ f_s(f_t(z)) = f_{s + t}(z), $$ with $f_0(z) = z$ and $f_1(z) = f(z).$
Alright, the case of $\sin z$ emphasizing the positive real axis is not terribly different, as long as we restrict to the interval $ 0 < x \leq \frac{\pi}{2}.$ For any such $x,$ define $x_0 = x, \; x_1 = \sin x, \; x_2 = \sin \sin x,$ and in general $ x_{n+1} = \sin x_n.$ This sequence approaches $0$, and in fact does so for any $z$ in a certain open set around the interval $ 0 < x \leq \frac{\pi}{2}$ that is called a petal.
Now, given a specific $x$ with $x_1 = \sin x$ and $ x_{n+1} = \sin x_n$ it is a result of Jean Ecalle at Orsay that we may take $$ \alpha(x) = \lim_{n \rightarrow \infty} \; \; \; \frac{3}{x_n^2} \; + \; \frac{6 \log x_n}{5} \; + \; \frac{79 x_n^2}{1050} \; + \; \frac{29 x_n^4}{2625} \; - \; n.$$
Note that $\alpha$ actually is defined on $ 0 < x < \pi$ with $\alpha(\pi - x) = \alpha(x),$ but the symmetry also means that the inverse function returns to the interval $ 0 < x \leq \frac{\pi}{2}.$
Before going on, the limit technique in the previous paragraph is given in pages 346-353 of Iterative Functional Equations by Marek Kuczma, Bogdan Choczewski, and Roman Ger. The solution is specifically Theorem 8.5.8 of subsection 8.5D, bottom of page 351 to top of page 353. Subsection 8.5A, pages 346-347, about Julia's equation, is part of the development.
As before, we define ( at least for $ 0 < x \leq \frac{\pi}{2}$) the parametrized interpolating functions, $$ f_s (x) = \alpha^{-1} \left( s + \alpha(x) \right) $$
In particular $$ f_{1/2} (x) = \alpha^{-1} \left( \frac{1}{2} + \alpha(x) \right) $$
I calculated all of this last night. First, by the kindness of Daniel Geisler, I have a pdf of the graph of this at:
http://zakuski.math.utsa.edu/~jagy/sine_half.pdf
Note that we use the evident symmetries $ f_{1/2} (-x) = - f_{1/2} (x)$ and $ f_{1/2} (\pi -x) = f_{1/2} (x)$
The result gives an interpolation of functions $f_s(x)$ ending at $ f_1(x)=\sin x$ but beginning at the continuous periodic sawtooth function, $x$ for $ -\frac{\pi}{2} \leq x \leq \frac{\pi}{2},$ then $\pi - x$ for $ \frac{\pi}{2} \leq x \leq \frac{3\pi}{2},$ continue with period $2 \pi.$ We do get $ f_s(f_t(z)) = f_{s + t}(z), $ plus the holomorphicity and symmetry of $\alpha$ show that $f_s(x)$ is analytic on the full open interval $ 0 < x < \pi.$
EDIT, TUTORIAL: Given some $z$ in the complex plane in the interior of the equilateral triangle with vertices at $0, \sqrt 3 + i, \sqrt 3 - i,$ take $z_0 = z, \; \; z_1 = \sin z, \; z_2 = \sin \sin z,$ in general $z_{n+1} = \sin z_n$ and $z_n = \sin^{[n]}(z).$ It does not take long to show that $z_n$ stays within the triangle, and that $z_n \rightarrow 0$ as $n \rightarrow \infty.$
Second, say $\alpha(z)$ is a true Fatou coordinate on the triangle, $\alpha(\sin z) = \alpha(z) + 1,$ although we do not know any specific value. Now, $\alpha(z_1) - 1 = \alpha(\sin z_0) - 1 = \alpha(z_0) + 1 - 1 = \alpha(z_0).$ Also $\alpha(z_2) - 2 = \alpha(\sin(z_1)) - 2 = \alpha(z_1) + 1 - 2 = \alpha(z_1) - 1 = \alpha(z_0).$ Induction, given $\alpha(z_n) - n = \alpha(z_0),$ we have $\alpha(z_{n+1}) - (n+1) = \alpha(\sin z_n) - n - 1 = \alpha(z_n) + 1 - n - 1 = \alpha(z_0).$
So, given $z_n = \sin^{[n]}(z),$ we have $\alpha(z_n) - n = \alpha(z).$
Third , let $L(z) = \frac{3}{z^2}+ \frac{6 \log z}{5} + \frac{79 z^2}{ 1050} + \frac{29 z^4}{2625}$. This is a sort of asymptotic expansion (at 0) for $\alpha(z),$ the error is $| L(z) - \alpha(z) | < c_6 |z|^6.$ It is unlikely that putting more terms on $L(z)$ leads to a convergent series, even in the triangle.
Fourth, given some $ z =z_0$ in the triangle. We know that $z_n \rightarrow 0$. So $| L(z_n) - \alpha(z_n) | < c_6 |z_n|^6.$ Or $| (L(z_n) - n ) - ( \alpha(z_n) - n) | < c_6 |z_n|^6 ,$ finally $$ | (L(z_n) - n ) - \alpha(z) | < c_6 |z_n|^6 .$$ Thus the limit being used is appropriate.
Fifth, there is a bootstrapping effect in use. We have no actual value for $\alpha(z),$ but we can write a formal power series for the solution of a Julia equation for $\lambda(z) = 1 / \alpha'(z),$ that is $\lambda(\sin z ) = \cos z \; \lambda(z).$ The formal power series for $\lambda(z)$ begins (KCG Theorem 8.5.1) with $- z^3 / 6,$ the first term in the power series of $\sin z$ after the initial $z.$ We write several more terms, $$\lambda(z) \asymp - \frac{z^3}{6} - \frac{z^5}{30} - \frac{41 z^7}{3780} - \frac{4 z^9}{945} \cdots.$$ We find the formal reciprocal, $$\frac{1}{\lambda(z)} = \alpha'(z) \asymp -\frac{6}{z^3} + \frac{6}{5 z} + \frac{79 z}{525} + \frac{116 z^3}{2625} + \frac{91543 z^5}{6063750}\cdots.$$ Finally we integrate term by term, $$\alpha(z) \asymp \frac{3}{z^2} + \frac{6 \log z }{5} + \frac{79 z^2}{1050} + \frac{29 z^4}{2625} + \frac{91543 z^6}{36382500}\cdots.$$ and truncate where we like, $$\alpha(z) = \frac{3}{z^2} + \frac{6 \log z }{5} + \frac{79 z^2}{1050} + \frac{29 z^4}{2625} + O(z^6)$$
Numerically, let me give some indication of what happens, in particular to emphasize $ f_{1/2} (\pi/2) = 1.140179\ldots.$