One small point...
The way the question is stated, there may be a slight ambiguity. One way (and almost certainly the intended way) to read the question is: given the (periodic) function $\sin^2(x)$, find its Fourier series on the interval $[0, \pi]$. In this case, $(1 - \cos(2x))/2$ is correct.
However, we could also read it as follows: given the function $\sin^2(x)$ defined on the interval $[0, \pi]$, find its Fourier series. In this case, we must first decide how to extend the function to be periodic. Of course the natural choice is to extend it to equal $\sin^2(x)$ for all $x$. Again, the cosine series is correct.
But... we do have the freedom to extend $\sin^2(x)$ to an odd function on $[-\pi, \pi]$ instead, in which case the Fourier series will contain only sine functions (with the coefficients computed in the usual way). The point being that there is in fact another series, featuring only sines, that converges to $\sin^2(x)$ on the interval $[0, \pi]$. Of course, the convergence isn't as fast ;-).
$f(x) = |\sin(x)| \quad \Rightarrow\quad f(x) = \left\{
\begin{array}{l l}
-\sin(x) & \quad \forall x \in [- \pi, 0\space]\\
\sin(x) & \quad \forall x \in [\space 0,\pi\space ]\\
\end{array} \right.$
The Fourier coefficients associated are
$$a_n= \frac{1}{\pi}\int_{-\pi}^\pi f(x) \cos(nx)\, dx = \frac{1}{\pi} \left[\int_{-\pi}^0 -\sin (x) \cos(nx)\, dx + \int_{0}^\pi \sin(x) \cos(nx)\, dx\right], \quad n \ge 0$$
$$b_n= \frac{1}{\pi}\int_{-\pi}^\pi f(x) \sin(nx)\, dx = \frac{1}{\pi} \left[\int_{-\pi}^0 -\sin (x) \sin(nx)\, dx + \int_{0}^\pi \sin(x) \sin(nx)\, dx\right], \quad n \ge 1$$
All functions are integrable so we can go on and compute the expressions for $a_n$ and $b_n$.
$$a_n = \cfrac{2 (\cos(\pi n)+1)}{\pi(1-n^2)}$$
$$b_n = 0$$
The $b_n = 0$ can be deemed obvious since the function $f(x) = |\sin(x)|$ is an even function. and $a_n$ could have been calculated as $\displaystyle a_n= \frac{2}{\pi}\int_{0}^\pi f(x) \cos(nx)\, dx $ only because the function is even.
The Fourier Series is $$\cfrac {a_0}{2} + \sum^{\infty}_{n=1}\left [ a_n \cos(nx) + b_n \sin (nx) \right ]$$
$$= \cfrac {2}{\pi}\left ( 1 + \sum^{\infty}_{n=1} \cfrac{(\cos(\pi n)+1)}{(1-n^2)}\cos(nx)\right )$$
$$= \cfrac {2}{\pi}\left ( 1 + \sum^{\infty}_{n=1} \cfrac{((-1)^n+1)}{(1-n^2)}\cos(nx)\right )$$
$$= \cfrac {2}{\pi}\left ( 1 + \sum^{\infty}_{n=1} \cfrac{2}{(1-4n^2)}\cos(2nx)\right )$$
Since for an odd $n$, $((-1)^n+1) = 0$ and for an even $n$, $((-1)^n+1) = 2$
At this point we can't just assume the function is equal to its Fourier Series, it has to satisfy certain conditions. See Convergence of Fourier series.
Without wasting time, (you still have to prove that it satisfies those conditions) we assume the Fourier Series converges to our function i.e
$$f(x) = |\sin(x)| = \cfrac {2}{\pi}\left ( 1 + \sum^{\infty}_{n=1} \cfrac{2}{(1-4n^2)}\cos(2nx)\right )$$
Note that $x=0$ gives $\cos(2nx) = 1$ then
$$f(0) = |\sin(0)| = \cfrac {2}{\pi}\left ( 1 + 2\sum^{\infty}_{n=1} \cfrac{1}{(1-4n^2)}\right ) =0$$ which implies that
$$\sum^{\infty}_{n=1} \cfrac{1}{(1-4n^2)} = \cfrac {-1}{2}$$ and $$\boxed {\displaystyle\sum^{\infty}_{n=1} \cfrac{1}{(4n^2 -1)}= -\sum^{\infty}_{n=1} \cfrac{1}{(1-4n^2)} = \cfrac {1}{2}}$$
Observe again that when $x = \cfrac \pi 2$, $\cos (2nx) = cos(n \pi) = (-1)^n$, thus
$$f \left (\cfrac \pi 2 \right) = \left |\sin \left (\cfrac \pi 2\right )\right | = \cfrac {2}{\pi}\left ( 1 + 2\sum^{\infty}_{n=1} \cfrac{(-1)^n}{(1-4n^2)}\right ) =1$$ which implies that
$$\sum^{\infty}_{n=1} \cfrac{(-1)^n}{(1-4n^2)} = \cfrac {1}{4}(\pi -2)$$ and $$\boxed {\displaystyle\sum^{\infty}_{n=1} \cfrac{(-1)^n}{(4n^2 -1)}= -\sum^{\infty}_{n=1} \cfrac{(-1)^n}{(1-4n^2)} = \cfrac {1}{4}(2-\pi)}$$
Best Answer
Development in cosine series means your function has to be even, so you define $f(x)=\sin(a|x|)$ on $[-\pi,\pi]$, and complete on $\Bbb R$ by periodicity. Then, your function is $2\pi$-periodic, continuous an piecewise $C^1$, hence the Dirichlet conditions apply.
Compute the Fourier coefficients:
$$a_n=\frac{1}{\pi}\int_{-\pi}^\pi \sin(a|x|)\cos(nx) \mathrm dx=\frac{2}{\pi}\int_{0}^\pi \sin(ax)\cos(nx) \mathrm dx$$
You have the identity $\sin(ax)\cos(nx)=\frac12[\sin(a+n)x+\sin(a-n)x]$, thus
$$a_n=\frac{1}{\pi}\int_{0}^\pi (\sin(a+n)x+\sin(a-n)x) \mathrm dx \\=\frac1\pi\left(\frac{1-\cos(a+n)\pi}{a+n}+\frac{1-\cos(a-n)\pi}{a-n}\right) \\=\frac{2a}{\pi(a^2-n^2)}\left[1-(-1)^n\cos(a\pi)\right]$$
For the last simplification, notice that $\cos(a+n)\pi=\cos(a-n)\pi=(-1)^n\cos(a\pi)$.
Then, for $x\in[0,\pi]$,
$$\sin (ax)=\frac{a_0}{2}+\sum_{n=1}^\infty a_n\cos (nx)=\frac{1-\cos(a\pi)}{a\pi}+\frac{2a}{\pi}\sum_{n=1}^\infty \frac{1-(-1)^n\cos(a\pi)}{a^2-n^2}\cos (nx)$$
Apart from the constant term, it's the same as your book's answer, in a slightly different form. Either you mistyped the answer, either there is a typo in your book.