This is what I did to find out the general solution :-
$$ (D^2 – 2D + 5)y = 10\sin x \\
\therefore \text{Auxiliary equation is } m^2 – 2m + 5 = 0 \\
\rightarrow m = \frac{-(-2) \pm \sqrt{(-2)^2 – 4.1.5}}{2.1} \\
= \frac{2 \pm \sqrt{4 – 20}}{2} \\
= \frac{2 \pm 4i}{2} = 1 \pm 2i \\
\therefore \text{Complementary function = } e^x(C_1\cos 2x + C_2 \sin 2x) \\
\text{Now, particular integral } = \frac{10 \sin x}{D^2 – 2D + 5} \\
= \frac{10 \sin x}{-1^2 -2D + 5} = \frac{10 \sin x}{4 – 2D} = \frac{5 \sin x}{2 – D} \\
= \frac{5\sin x}{2(1-\frac{D}{2})} = \frac{5}{2}\left(1-\frac{D}{2}\right)^{-1}\sin x \\
= \frac{5}{2}\left(1 + \frac{D}{2} + …\right)\sin x \\
= \frac{5}{2}\left(\sin x + \frac{D}{2}\sin x\right) = \frac{5}{2}\left(\sin x + \frac{\cos x}{2}\right) = \frac{5}{4}(2\sin x + \cos x) \\
\text{Now, } y = \text{CF + PI} \\
\therefore y = e^x(C_1\cos 2x + C_2 \sin 2x) + \frac{5}{4}(2\sin x + \cos x) $$
But the answer given in the book is :-
$$ y = e^x(C_1 \cos 2x + C_2 \sin 2x) + 2 \sin x + \cos x $$
Best Answer
$PI = \frac{1 }{D^2-2D+5}(10\sin x) = 10\frac{1}{4-2D}\sin x = \frac{5}{2-D}{IP e^{ix}}$
$PI = IP\frac{5}{2-i}e^{ix} = 5IP (\frac{2}{5}+\frac{i}{5})(\cos x +i\sin x)$
$PI = IP (2\cos x +i\cos x +2i\sin x - \sin x) = \cos x + 2\sin x $
You can't simply write $(1-\frac{D}{2})^{-1} = (1+\frac{D}{2}\cdots)$ because $D^2(\sin x) , D^3(\sin x) \cdots $ exist.
If it were simply $x^n$, then we can take up to only $D^n(x^n)$ and take $D^{n+1}(x^n) = D^{n+2}(x^n) =\cdots = 0$, but not in this case.