Here's how I would motivate the solution:
First, the problem will be VASTLY simpler if you replace $\cos(nx)$ with one half the sum of a positive and negative complex exponential and $\sin(nx)$ as one half the difference between the positive and negative complex exponential, and then once the two series are expressed in terms of complex exponentials, then multiplication of Fourier terms becomes easy.
That's because the coefficient of the $m$-th term in the product will contain terms from the two multiplier series whose indices add to $m$, such as $(m-2,2)$, $(m-1,1)$, $(m,0)$, $(m+1,-1)$, $(m+2,-2)$, and so on, where the first index in each pair refers to the first series and the second index refers to the second series, and this is where the convolution on indices comes from.
$$\text{a}_{0}=\frac{1}{\pi}\int_{-\pi}^{\pi}(\sin(x)+\cos(2x))\space\text{d}x=$$
$$\frac{1}{\pi}\left[\int_{-\pi}^{\pi}\sin(x)\space\text{d}x+\int_{-\pi}^{\pi}\cos(2x)\space\text{d}x\right]=$$
Now, use: $\int\sin(x)\space\text{d}x=\text{C}-\cos(x)$
$$\frac{1}{\pi}\left[\left[-\cos(x)\right]_{-\pi}^{\pi}+\int_{-\pi}^{\pi}\cos(2x)\space\text{d}x\right]=$$
Substitute $u=2x$ and $\text{d}u=2\space\text{d}x$.
This gives a new lower bound $u=-2\pi$ and upper bound $u=2\pi$:
$$\frac{1}{\pi}\left[\left[-\cos(x)\right]_{-\pi}^{\pi}+\frac{1}{2}\int_{-2\pi}^{2\pi}\cos(u)\space\text{d}u\right]=$$
Now, use: $\int\cos(x)\space\text{d}x=\sin(x)+\text{C}$
$$\frac{1}{\pi}\left[\left[-\cos(x)\right]_{-\pi}^{\pi}+\frac{1}{2}\left[\sin(u)\right]_{-2\pi}^{2\pi}\right]=0$$
NEXT:
$$\text{a}_{n}=\frac{1}{\pi}\int_{-\pi}^{\pi}(\sin(x)+\cos(2x))\cos(nx)\space\text{d}x=$$
$$\frac{1}{\pi}\left[\int_{-\pi}^{\pi}\sin(x)\cos(nx)\space\text{d}x+\int_{-\pi}^{\pi}\cos(2x)\cos(nx)\space\text{d}x\right]=$$
Now, as you noticed already:
Since $\sin(x)\cos(nx)$ is an odd function and the interval $[\pi,\pi]$ is symmetric about $0$, so:
$$\int_{-\pi}^{\pi}\sin(x)\cos(nx)\space\text{d}x=0$$
$$\frac{1}{\pi}\int_{-\pi}^{\pi}\cos(2x)\cos(nx)\space\text{d}x=$$
Use the trigonometric identity $\cos(a)\cos(b)=\frac{\cos(a-b)+cos(a+b)}{2}$:
$$\frac{1}{2\pi}\int_{-\pi}^{\pi}(\cos(x(n-2))+\cos(x(n+2)))\space\text{d}x=$$
$$\frac{1}{2\pi}\left[\int_{-\pi}^{\pi}\cos(x(n+2))\space\text{d}x+\int_{-\pi}^{\pi}\cos(x(n-2))\space\text{d}x\right]=$$
For the left integral:
Substitute $u=x(n+2)$ and $\text{d}u=(n+2)\space\text{d}x$.
This gives a new lower bound $u=(-\pi)(n+2)=-\pi(n+2)$ and upper bound $u=\pi(n+2)$:
$$\frac{1}{2\pi}\left[\frac{1}{n+2}\int_{-\pi(n+2)}^{\pi(n+2)}\cos(u)\space\text{d}u+\int_{-\pi}^{\pi}\cos(x(n-2))\space\text{d}x\right]=$$
For the right integral:
Substitute $s=x(n-2)$ and $\text{d}s=(n-2)\space\text{d}x$.
This gives a new lower bound $s=(-\pi)(n-2)=-\pi(n-2)$ and upper bound $s=\pi(n-2)$:
$$\frac{1}{2\pi}\left[\frac{1}{n+2}\int_{-\pi(n+2)}^{\pi(n+2)}\cos(u)\space\text{d}u+\frac{1}{n-2}\int_{-\pi(n-2)}^{\pi(n-2)}\cos(s)\space\text{d}s\right]=$$
Now, use: $\int\cos(x)\space\text{d}x=\sin(x)+\text{C}$
$$\frac{1}{2\pi}\left[\frac{1}{n+2}\left[\sin(u)\right]_{-\pi(n+2)}^{\pi(n+2)}+\frac{1}{n-2}\left[\sin(s)\right]_{-\pi(n-2)}^{\pi(n-2)}\right]=$$
$$\frac{\sin(\pi n)}{\pi(n+2)}+\frac{\sin(\pi n)}{\pi(n-2)}=\frac{2n\sin(\pi n)}{\pi(n^2-4)}$$
NEXT:
$$\text{b}_{n}=\frac{1}{\pi}\int_{-\pi}^{\pi}(\sin(x)+\cos(2x))\sin(nx)\space\text{d}x=$$
$$\frac{1}{\pi}\left[\int_{-\pi}^{\pi}\sin(x)\sin(nx)\space\text{d}x+\int_{-\pi}^{\pi}\cos(2x)\sin(nx)\space\text{d}x\right]=$$
Now, as you noticed already:
Since $\cos(2x)\sin(nx)$ is an odd function and the interval $[\pi,\pi]$ is symmetric about $0$, so:
$$\int_{-\pi}^{\pi}\cos(2x)\sin(nx)\space\text{d}x=0$$
$$\frac{1}{\pi}\int_{-\pi}^{\pi}\sin(x)\sin(nx)\space\text{d}x=$$
Use the trigonometric identity $\sin(a)\sin(b)=\frac{\cos(a-b)-cos(a+b)}{2}$:
$$\frac{1}{2\pi}\int_{-\pi}^{\pi}(\cos(x(n-1))-\cos(x(n+1)))\space\text{d}x=$$
$$\frac{1}{2\pi}\left[\int_{-\pi}^{\pi}\cos(x(n-1))\space\text{d}x-\int_{-\pi}^{\pi}\cos(x(n+1))\space\text{d}x\right]=$$
For the right integral:
Substitute $u=x(n+1)$ and $\text{d}u=(n+1)\space\text{d}x$.
This gives a new lower bound $u=(-\pi)(n+1)=-\pi(n+1)$ and upper bound $u=\pi(n+1)$:
$$\frac{1}{2\pi}\left[\int_{-\pi}^{\pi}\cos(x(n-1))\space\text{d}x-\frac{1}{n+1}\int_{-\pi(n+1)}^{\pi(n+1)}\cos(u)\space\text{d}u\right]=$$
For the left integral:
Substitute $s=x(n-1)$ and $\text{d}s=(n-1)\space\text{d}x$.
This gives a new lower bound $s=(-\pi)(n-1)=-\pi(n-1)$ and upper bound $s=\pi(n-1)$:
$$\frac{1}{2\pi}\left[\frac{1}{n-1}\int_{-\pi(n-1)}^{\pi(n-1)}\cos(s)\space\text{d}s-\frac{1}{n+1}\int_{-\pi(n+1)}^{\pi(n+1)}\cos(u)\space\text{d}u\right]=$$
Now, use: $\int\cos(x)\space\text{d}x=\sin(x)+\text{C}$
$$\frac{1}{2\pi}\left[\frac{1}{n-1}\left[\sin(s)\right]_{-\pi(n-1)}^{\pi(n-1)}-\frac{1}{n+1}\left[\sin(u)\right]_{-\pi(n+1)}^{\pi(n+1)}\right]=$$
$$\frac{\sin(\pi n)}{\pi(1-n)}+\frac{\sin(\pi n)}{\pi(1+n)}=\frac{2\sin(\pi n)}{\pi(1-n^2)}$$
CONCLUSION:
- $$\text{a}_0=0$$
- $$\text{a}_n=\frac{2n\sin(\pi n)}{\pi(n^2-4)}$$
- $$\text{b}_n=\frac{2\sin(\pi n)}{\pi(1-n^2)}$$
Best Answer
I see someone else already answered it in your comments, but one way to remember it is $$\frac{2}{\pi}\int\limits_0^\pi \sin(mx)\sin(nx)dx=\delta_{mn}$$
Where $\delta_{mn}$ is 1 if $m=n$ and 0 otherwise.
A way of thinking about it in Linear Algebra terms is that the functions $\sin(nx)$ are basis vectors in a space with $n$ vectors ($\sin(x)$,$\sin(2x)$, $\sin(3x)$, etc...).
All of those vectors are mutually perpendicular, so when you take that integral it's like asking the dot product between those two vectors (dot product and this integral are actually related). In this case every vector is mutually perpendicular except the vector and itself.