$\newcommand{\+}{^{\dagger}}
\newcommand{\angles}[1]{\left\langle #1 \right\rangle}
\newcommand{\braces}[1]{\left\lbrace #1 \right\rbrace}
\newcommand{\bracks}[1]{\left\lbrack #1 \right\rbrack}
\newcommand{\ceil}[1]{\,\left\lceil #1 \right\rceil\,}
\newcommand{\dd}{{\rm d}}
\newcommand{\down}{\downarrow}
\newcommand{\ds}[1]{\displaystyle{#1}}
\newcommand{\equalby}[1]{{#1 \atop {= \atop \vphantom{\huge A}}}}
\newcommand{\expo}[1]{\,{\rm e}^{#1}\,}
\newcommand{\fermi}{\,{\rm f}}
\newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,}
\newcommand{\half}{{1 \over 2}}
\newcommand{\ic}{{\rm i}}
\newcommand{\iff}{\Longleftrightarrow}
\newcommand{\imp}{\Longrightarrow}
\newcommand{\isdiv}{\,\left.\right\vert\,}
\newcommand{\ket}[1]{\left\vert #1\right\rangle}
\newcommand{\ol}[1]{\overline{#1}}
\newcommand{\pars}[1]{\left( #1 \right)}
\newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
\newcommand{\pp}{{\cal P}}
\newcommand{\root}[2][]{\,\sqrt[#1]{\vphantom{\large A}\,#2\,}\,}
\newcommand{\sech}{\,{\rm sech}}
\newcommand{\sgn}{\,{\rm sgn}}
\newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}}
\newcommand{\ul}[1]{\underline{#1}}
\newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert}$
$\ds{I \equiv
{1 \over \pi}\int_{0}^{\pi}\theta^{2}\ln^{2}\pars{2\cos\pars{\theta \over 2}}
\,\dd\theta = {11\pi^{4} \over 180} = {11 \over 2}\,\zeta\pars{4}}$
\begin{align}
I&={1 \over 2\pi}\int_{-\pi}^{\pi}\theta^{2}
\ln^{2}\pars{2\root{1 + \cos\pars{\theta} \over 2}}\,\dd\theta
\\[3mm]&={1 \over 2\pi}
\int_{\verts{z} = 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}}
\bracks{-\ln^{2}\pars{z}}\ln^{2}\pars{\root{2}\root{1 + {z^{2} + 1 \over 2z}}}\,
{\dd z \over \ic z}
\\[3mm]&={\ic \over 2\pi}
\int_{\verts{z} = 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}}
\ln^{2}\pars{z}\ln^{2}\pars{z + 1 \over z^{1/2}}\,{\dd z \over z}
\\[3mm]&={\ic \over 2\pi}\lim_{\mu \to -1 \atop \nu \to 0}
\partiald[2]{}{\mu}\partiald[2]{}{\nu}
\int_{\verts{z} = 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}}
z^{\mu}\pars{z + 1 \over z^{1/2}}^{\nu}\,\dd z
\\[3mm]&={\ic \over 2\pi}\lim_{\mu \to -1 \atop \nu \to 0}
\partiald[2]{}{\mu}\partiald[2]{}{\nu}
\int_{\verts{z} = 1 \atop {\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi}}z^{\mu - \nu/2}\pars{z + 1}^{\nu}\,\dd z
\qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\pars{1}
\end{align}
The integration in $\pars{1}$ is given by:
\begin{align}
&\int_{\verts{z} = 1}
z^{\mu - \nu/2}\pars{z + 1}^{\nu}\,\dd z
\\[3mm]&=-\int_{-1}^{0}\pars{-x}^{\mu - \nu/2}\expo{\ic\pi\pars{\mu - \nu/2}}
\pars{x + 1}^{\nu}\,\dd x
-\int_{0}^{-1}\pars{-x}^{\mu - \nu/2}\expo{-\ic\pi\pars{\mu - \nu/2}}
\pars{x + 1}^{\nu}\,\dd x
\\[3mm]&=-\expo{\ic\pi\pars{\mu - \nu/2}}\int_{0}^{1}x^{\mu - \nu/2}
\pars{-x + 1}^{\nu}\,\dd x
+\expo{-\ic\pi\pars{\mu - \nu/2}}\int_{0}^{1}x^{\mu - \nu/2}
\pars{-x + 1}^{\nu}\,\dd x
\\[3mm]&=2\ic\sin\pars{\pi\bracks{{\nu \over 2} - \mu}}
{\rm B}\pars{\mu - {\nu \over 2} + 1,\nu + 1}\tag{2}
\end{align}
where $\ds{{\rm B}\pars{x,y} = \int_{0}^{1}t^{x - 1}\pars{1 - t}^{y - 1}\,\dd t}$,
$\ds{\pars{~\mbox{with}\ \Re\pars{x} > 0,\ \Re\pars{y} > 0~}}$ is the
Beta Function.
With $\pars{1}$ and $\pars{2}$, $\ds{I}$ is reduced to:
$$
I=-\,{1 \over \pi}\lim_{\mu \to -1 \atop \nu \to 0}
\partiald[2]{}{\mu}\partiald[2]{}{\nu}\bracks{%
\sin\pars{\pi\bracks{{\nu \over 2} - \mu}}
{\rm B}\pars{\mu - {\nu \over 2} + 1,\nu + 1}}
$$
Since $\ds{{\rm B}\pars{x,y}=
{\Gamma\pars{x}\Gamma\pars{y} \over \Gamma\pars{x + y}}}$
( $\ds{\Gamma\pars{z}}$ is the
GammaFunction ):
\begin{align}
I&=-\,{1 \over \pi}\
\lim_{\mu \to -1 \atop \nu \to 0}
\partiald[2]{}{\mu}\partiald[2]{}{\nu}\bracks{%
\sin\pars{\pi\bracks{{\nu \over 2} - \mu}}
{\Gamma\pars{\mu - \nu/2 + 1}\Gamma\pars{\nu + 1}
\over \Gamma\pars{\mu + \nu/2 + 2}}}
\\[3mm]&=-\
\overbrace{\lim_{\mu \to -1 \atop \nu \to 0}
\partiald[2]{}{\mu}\partiald[2]{}{\nu}\bracks{%
{\Gamma\pars{\nu + 1}
\over \Gamma\pars{\nu/2 - \mu}\Gamma\pars{\mu + \nu/2 + 2}}}}
^{\ds{=\ -\,{11\pi^{4} \over 180}}}
\end{align}
where we used the identity
$\ds{\Gamma\pars{z}\Gamma\pars{1 - z} = {\pi \over \sin\pars{\pi z}}}$
Then,
$$
I \equiv
\color{#00f}{\large%
{1 \over \pi}\int_{0}^{\pi}\theta^{2}\ln^{2}\pars{2\cos\pars{\theta \over 2}} \,\dd\theta}
= \color{#00f}{\large{11\pi^{4} \over 180} = {11 \over 2}\,\zeta\pars{4}}
$$
To evaluate this integral using a quarter-circle contour in the first quadrant of the complex plane (that is indented at the origin), we can exploit the fact that if we're in the first quadrant , then $$ \left|\exp \left(\color{red}{-}\frac{i}{z^{2}} \right) \right| = \left|\exp \left(\color{red}{-}\frac{i}{(x+iy)^2} \right) \right| = \exp \left(\color{red}{-}\frac{2xy}{(x^{2}+y^{2})^{2}} \right) \le 1.$$
In other words, the magnitude of $\exp \left(\color{red}{-}\frac{i}{z^{2}} \right)$ remains bounded in we stay in the first quadrant, which includes near the essential singularity at the origin.
Notice the negative sign. This is not true for $\exp \left(\frac{i}{z^{2}} \right)$, and thus not true for $\cos \left(\frac{1}{z^{2}} \right).$
Therefore, if we integrate $$f(z) = \frac{z\exp\left(\color{red}{-}\frac{i}{z^{2}}\right)}{z^{4}+4} $$ around the contour, the integral along the big arc and the integral along the small arc will both vanish in the limit.
We're then left with $$\begin{align} \int_{0}^{\infty} \frac{x \exp \left(\color{red}{-}\frac{i}{x^{2}}\right)}{x^{4}+4} \, \mathrm dx + \int_{\infty}^{0} \frac{(it) \exp \left( \frac{i}{t^{2}}\right)}{t^{4}+4} \, (i \, \mathrm dt) &= 2 \int_{0}^{\infty} \frac{x \cos \left(\frac{1}{x^{2}} \right)}{x^{4}+4} \, \ \mathrm dx \\ &= 2 \pi i \operatorname{Res}[f(z),1+i] \\ & = \frac{\pi}{4 \sqrt{e}}. \end{align} $$
I asked a question about how to deal with essential singularities that are on the contour several years ago.
See here.
EDIT:
If we wanted to integrate the function $\frac{z\exp\left(\frac{i}{z^{2}}\right)}{z^{4}+4}$ instead, we could use a quarter-circle in the fourth quadrant.
Best Answer
If you wish to exploit the residue theorem, then first exploit the fact that the integral is even. In addition, use Euler's formula to write $2\cos(2\theta)+\cos(3\theta)=\text{Re}(2e^{i2\theta}+e^{i3\theta})$.
Then, we have
$$\begin{align} \int_0^\pi \frac{2\cos(2\theta)+\cos(3\theta)}{5+4\cos(\theta)}\,d\theta&=\frac12\text{Re}\left(\oint_{|z|=1}\frac{2z^2+z^3}{5+2(z+z^{-1})}\,\frac{1}{iz}\,dz\right)\\\\ &=\frac12\text{Re}\left(\frac1i\oint_{|z|=1}\frac{z^2(2+z)}{(2z+1)(z+2)}\,dz\right)\\\\ &=\frac12\text{Re}\left(\frac1i\oint_{|z|=1}\frac{z^2}{2z+1}\,dz\right)\\\\ &=\frac{\pi}{8} \end{align}$$
And we are done!