As @LutzL said in the comments, use:
$$\cos^2\left(x\right)=\frac{1+\cos\left(2x\right)}{2}\tag1$$
And you're correct.
Another approach, use Laplace transform:
$$\mathcal{L}_x\left[\text{y}''\left(x\right)+\text{y}\left(x\right)\right]_{\left(\text{s}\right)}=\mathcal{L}_x\left[\cos^2\left(x\right)\right]_{\left(\text{s}\right)}=\frac{1}{2}\cdot\left(\mathcal{L}_x\left[1\right]_{\left(\text{s}\right)}+\mathcal{L}_x\left[\cos\left(2x\right)\right]_{\left(\text{s}\right)}\right)\tag2$$
So, we use that:
- $$\mathcal{L}_x\left[\text{y}''\left(x\right)\right]_{\left(\text{s}\right)}=\text{s}^2\cdot\text{Y}\left(\text{s}\right)-\text{s}\cdot\text{y}\left(0\right)-\text{y}'\left(0\right)\tag3$$
- $$\mathcal{L}_x\left[\text{y}\left(x\right)\right]_{\left(\text{s}\right)}=\text{Y}\left(\text{s}\right)\tag4$$
- $$\mathcal{L}_x\left[1\right]_{\left(\text{s}\right)}=\frac{1}{\text{s}}\tag5$$
- $$\mathcal{L}_x\left[\cos\left(2x\right)\right]_{\left(\text{s}\right)}=\frac{\text{s}}{4+\text{s}^2}\tag6$$
So, we get:
$$\text{s}^2\cdot\text{Y}\left(\text{s}\right)-\text{s}\cdot\text{y}\left(0\right)-\text{y}'\left(0\right)+\text{Y}\left(\text{s}\right)=\frac{1}{2}\cdot\left(\frac{1}{\text{s}}+\frac{\text{s}}{4+\text{s}^2}\right)\tag7$$
Solving for $\text{Y}\left(\text{s}\right)$:
$$\text{Y}\left(\text{s}\right)=\frac{\frac{1}{2}\cdot\left(\frac{1}{\text{s}}+\frac{\text{s}}{4+\text{s}^2}\right)+\text{s}\cdot\text{y}\left(0\right)+\text{y}'\left(0\right)}{1+\text{s}^2}\tag8$$
Now, using inverse Laplace transform:
$$\text{y}\left(x\right)=\frac{3-\cos\left(2x\right)+\cos\left(x\right)\left(6\text{y}\left(0\right)-2\right)+6\text{y}'\left(0\right)\sin\left(x\right)}{6}\tag9$$
Best Answer
What you have done so far is correct. You should proceed as follows:
Write the last two equations as a system
$$\left(\begin{array}{cc} e^{2x} & e^{-2x} \\ e^{2x} & -e^{-2x} \end{array}\right) \cdot \left(\begin{array}{c} C_{1}^{\prime} \\ C_{2}^{\prime} \end{array}\right)=\left(\begin{array}{c} 0 \\ \tan(2x) \end{array}\right)$$ Invert the matrix on the LHS to get $$\left( \begin{array}{cc} C_{1}^{\prime} \\ C_{2}^{\prime} \end{array}\right)=-\frac{1}{2}\left(\begin{array}{cc} -e^{-2x} & -e^{-2x} \\ -e^{2x} & e^{2x}\end{array} \right)\cdot \left( \begin{array}{c} 0 \\ \tan(2x) \end{array}\right)=-\frac{1}{2}\left(\begin{array}{c} -e^{-2x}\tan(2x) \\ e^{2x}\tan(2x) \end{array} \right)$$ This gives you a system of first order equations which you can solve by integrating. However, its not a particularly nice integral.