To calculate these two sums, we are going to establish two relations and solve them by elimination.
To establish the first relation, we use $\displaystyle I=\int_0^1\frac{\ln^4(1+x)+6\ln^2(1-x)\ln^2(1+x)}{x}\ dx=\frac{21}4\zeta(5)\tag{1}$
which was proved by Khalef Ruhemi ( unfortunately he is not an MSE user).
The proof as follows: using the algebraic identity $\ b^4+6a^2b^2=\frac12(a-b)^4+\frac12(a+b)^4-a^4$
with $\ a=\ln(1-x)$ and $\ b=\ln(1+x)$ , divide both sides by $x$ then integrate, we get
$$I=\frac12\underbrace{\int_0^1\frac1x{\ln^4\left(\frac{1-x}{1+x}\right)}\ dx}_{\frac{1-x}{1+x}=y}+\underbrace{\frac12\int_0^1\frac{\ln^4(1-x^2)}{x}\ dx}_{x^2=y}-\int_0^1\frac{\ln^4(1-x)}{x}\ dx$$
$$=\int_0^1\frac{\ln^4x}{1-x^2}+\frac14\int_0^1\frac{\ln^4(1-x)}{x}\ dx-\int_0^1\frac{\ln^4(1-x)}{x}\ dx$$
$$=\frac12\int_0^1\frac{\ln^4x}{1-x}+\frac12\int_0^1\frac{\ln^4x}{1+x}-\frac34\underbrace{\int_0^1\frac{\ln^4(1-x)}{x}\ dx}_{1-x=y}$$
$$=\frac12\int_0^1\frac{\ln^4x}{1+x}\ dx+\frac14\int_0^1\frac{\ln^4x}{1-x}\ dx=\frac12\left(\frac{45}{2}\zeta(5)\right)+\frac14(24\zeta(5))=\frac{21}4\zeta(5)$$
On the other hand, $\quad\displaystyle I=\underbrace{\int_0^1\frac{\ln^4(1+x)}{x}\ dx}_{I_1}+6\int_0^1\frac{\ln^2(1-x)\ln^2(1+x)}{x}\ dx$
Using $\ln^2(1+x)=2\sum_{n=1}^\infty(-1)^n\left(\frac{H_n}{n}-\frac{1}{n^2}\right)x^n\ $ for the second integral, we get
\begin{align}
I&=I_1+12\sum_{n=1}^\infty(-1)^n\left(\frac{H_n}{n}-\frac{1}{n^2}\right)\int_0^1x^{n-1}\ln^2(1-x)\ dx\\
I&=I_1+12\sum_{n=1}^\infty(-1)^n\left(\frac{H_n}{n}-\frac{1}{n^2}\right)\left(\frac{H_n^2+H_n^{(2)}}{n}\right)\\
I&=I_1+12\sum_{n=1}^\infty(-1)^n\left(\frac{H_n^3+H_nH_n^{(2)}}{n^2}\right)-12\sum_{n=1}^\infty(-1)^n\left(\frac{H_n^2+H_n^{(2)}}{n^3}\right)\tag{2}
\end{align}
From $(1)$ and $(2)$, we get
$$\boxed{\small{R_1=\sum_{n=1}^\infty\frac{(-1)^nH_n^3}{n^2}+\sum_{n=1}^\infty\frac{(-1)^nH_nH_n^{(2)}}{n^2}=\frac{7}{16}\zeta(5)+\sum_{n=1}^\infty\frac{(-1)^nH_n^2}{n^3}+\sum_{n=1}^\infty\frac{(-1)^nH_n^{(2)}}{n^3}-\frac{1}{12}I_1}}$$
and the first relation is established.
To get the second relation, we need to use the sterling number formula ( check here)
$$ \frac{\ln^k(1-x)}{k!}=\sum_{n=k}^\infty(-1)^k \begin{bmatrix} n \\ k \end{bmatrix}\frac{x^n}{n!}$$
letting $k=4$ and using $\displaystyle\begin{bmatrix} n \\ 4 \end{bmatrix}=\frac{1}{3!}(n-1)!\left[\left(H_{n-1}\right)^3-3H_{n-1}H_{n-1}^{(2)}+2H_{n-1}^{(3)}\right],$ we get $$\frac14\ln^4(1-x)=\sum_{n=1}^\infty\frac{x^{n+1}}{n+1}\left(H_n^3-3H_nH_n^{(2)}+2H_n^{(3)}\right)$$
differentiate both sides with respect to $x$, we get
$$-\frac{\ln^3(1-x)}{1-x}=\sum_{n=1}^\infty x^n\left(H_n^3-3H_nH_n^{(2)}+2H_n^{(3)}\right)$$
Now replace $x$ with $-x$ then multiply both sides by $\frac{\ln x}{x}$ and integrate, we get
$$-\sum_{n=1}^\infty(-1)^n\left(H_n^3-3H_nH_n^{(2)}+2H_n^{(3)}\right)\int_0^1x^{n-1}\ln x\ dx=\int_0^1\frac{\ln^3(1+x)\ln x}{x(1+x)}\ dx$$
$$\sum_{n=1}^\infty \frac{(-1)^n}{n^2}\left(H_n^3-3H_nH_n^{(2)}+2H_n^{(3)}\right)=\int_0^1\frac{\ln^3(1+x)\ln x}{x}\ dx-\underbrace{\int_0^1\frac{\ln^3(1+x)\ln x}{1+x}\ dx}_{IBP}$$
$$\sum_{n=1}^\infty \frac{(-1)^n}{n^2}\left(H_n^3-3H_nH_n^{(2)}+2H_n^{(3)}\right)=\int_0^1\frac{\ln^3(1+x)\ln x}{x}\ dx+\frac14I_1$$
Rearranging the terms, we get
$$\boxed{R_2=\sum_{n=1}^\infty\frac{(-1)^nH_n^3}{n^2}-3\sum_{n=1}^\infty\frac{(-1)^nH_nH_n^{(2)}}{n^2}=\int_0^1\frac{\ln^3(1+x)\ln x}{x}-2\sum_{n=1}^\infty\frac{(-1)^nH_n^{(3)}}{n^2}+\frac14I_1}$$
and the second relation is established.
Now we are ready to calculate the first sum.
\begin{align}
\sum_{n=1}^\infty\frac{(-1)^nH_n^3}{n^2}&=\frac{3R_1+R_2}{4}\\
&=\frac34\sum_{n=1}^\infty\frac{(-1)^nH_n^2}{n^3}+\frac34\sum_{n=1}^\infty\frac{(-1)^nH_n^{(2)}}{n^3}-\frac12\sum_{n=1}^\infty\frac{(-1)^nH_n^{(3)}}{n^2}\\
&\quad+\frac14\int_0^1\frac{\ln x\ln^3(1+x)}{x}\ dx+\frac{21}{64}\zeta(5)
\end{align}
the closed form of the first and second sum can be found here and the closed form of the third sum is evaluated here. as for the integral, I evaluated it here.
by combining these results, we get our closed form.
and the second sum.
$$\sum_{n=1}^\infty\frac{(-1)^nH_nH_n^{(2)}}{n^2}=\frac{R_1-R_2}{4}$$
$$\small{=\frac14\sum_{n=1}^\infty\frac{(-1)^nH_n^2}{n^3}+\frac14\sum_{n=1}^\infty\frac{(-1)^nH_n^{(2)}}{n^3}+\frac12\sum_{n=1}^\infty\frac{(-1)^nH_n^{(3)}}{n^2}-\frac14\int_0^1\frac{\ln x\ln^3(1+x)}{x}\ dx-\frac1{12}I_1+\frac{7}{64}\zeta(5)}$$
lets calculate $I_1$ and by setting $\frac1{1+x}=y$, we get
\begin{align}
I_1&=\int_0^1\frac{\ln^4(1+x)}{x}=\int_{1/2}^1\frac{\ln^4x}{x}\ dx+\int_{1/2}^1\frac{\ln^4x}{1-x}\ dx\\
&=\frac15\ln^52+\sum_{n=1}^\infty\int_{1/2}^1 x^{n-1}\ln^4x\ dx\\
&=\frac15\ln^52+\sum_{n=1}^\infty\left(\frac{24}{n^5}-\frac{24}{n^52^n}-\frac{24\ln2}{n^42^n}-\frac{12\ln^22}{n^32^n}-\frac{4\ln^32}{n^22^n}-\frac{\ln^42}{n2^n}\right)\\
&=4\ln^32\zeta(2)-\frac{21}2\ln^22\zeta(3)+24\zeta(5)-\frac45\ln^52-24\ln2\operatorname{Li}_4\left(\frac12\right)-24\operatorname{Li}_5\left(\frac12\right)
\end{align}
by combining the result of $I_1$ along with the results we used in our first sum, we get the closed form of the second sum.
UPDATE:
The identity used above:
$$-\frac{\ln^3(1-x)}{1-x}=\sum_{n=1}^\infty x^n\left(H_n^3-3H_nH_n^{(2)}+2H_n^{(3)}\right)$$
can also be proved this way.
A solution using Abel's summation as suggested by Cornel.
Let $\ \displaystyle S\ $ denote $\ \displaystyle \sum_{k=1}^\infty\frac{H_k^{(2)}}{(2k+1)^2}\ $
and by using Abel's summation:
$\displaystyle\sum_{k=1}^n a_k b_k=A_nb_{n+1}+\sum_{k=1}^{n}A_k\left(b_k-b_{k+1}\right)\ $ where $\ \displaystyle A_n=\sum_{i=1}^n a_i\ $
and letting let $\ \displaystyle a_k=\frac{1}{(2k+1)^2}\ $ , $\ \displaystyle b_k=H_k^{(2)}$, we get
\begin{align}
\sum_{k=1}^n\frac{H_k^{(2)}}{(2k+1)^2}&=\sum_{i=1}^n\frac{H_{n+1}^{(2)}}{(2i+1)^2}-\sum_{k=1}^n\frac{1}{(k+1)^2}\left(\sum_{i=1}^k\frac{1}{(2i+1)^2}\right)\\
&=\sum_{i=1}^n\frac{H_{n+1}^{(2)}}{(2i+1)^2}-\sum_{k=1}^n\frac{1}{(k+1)^2}\left(H_{2k}^{(2)}-\frac14H_{k}^{(2)}+\frac{1}{(2k+1)^2}-1\right)
\end{align}
Letting $n$ approach $\infty$, we get
\begin{align}
S&=\zeta(2)\sum_{i=1}^\infty\frac{1}{(2i+1)^2}-\sum_{k=1}^\infty\frac{1}{(k+1)^2}\left(H_{2k}^{(2)}-\frac14H_{k}^{(2)}\right)\\
&\quad-\sum_{k=1}^\infty\frac{1}{(k+1)^2(2k+1)^2}+\sum_{k=1}^\infty\frac1{(k+1)^2}\\
&=\zeta(2)\left(\frac34\zeta(2)-1\right)-\sum_{k=1}^\infty\frac{1}{k^2}\left(H_{2k}^{(2)}-\frac14H_{k}^{(2)}-\frac{1}{(2k-1)^2}\right)\\
&\quad-\sum_{k=1}^\infty\frac{1}{(k+1)^2(2k+1)^2}+\zeta(2)-1\\
&=\frac{15}8\zeta(4)-1-\sum_{k=1}^\infty\frac{1}{k^2}\left(H_{2k}^{(2)}-\frac14H_{k}^{(2)}\right)+\sum_{k=1}^\infty\frac{1}{k^2(2k-1)^2}-\sum_{k=1}^\infty\frac{1}{(k+1)^2(2k+1)^2}\\
&=\frac{15}8\zeta(4)-1-\sum_{k=1}^\infty\frac{1}{k^2}\left(H_{2k}^{(2)}-\frac14H_{k}^{(2)}\right)+1\\
&\quad+\sum_{k=1}^\infty\frac{1}{(k+1)^2(2k+1)^2}-\sum_{k=1}^\infty\frac{1}{(k+1)^2(2k+1)^2}\\
&=\frac{15}8\zeta(4)-\sum_{k=1}^\infty\frac{1}{k^2}\left(H_{2k}^{(2)}-\frac14H_{k}^{(2)}\right)\\
&=\frac{15}8\zeta(4)-4\sum_{k=1}^\infty\frac{H_{2k}^{(2)}}{(2k)^2}+\frac14\sum_{k=1}^\infty\frac{H_k^{(2)}}{k^2}\\
&=\frac{15}8\zeta(4)-4\left(\frac12\sum_{k=1}^\infty\frac{H_{k}^{(2)}}{k^2}+\frac12\sum_{k=1}^\infty\frac{(-1)^kH_k^{(2)}}{k^2}\right)+\frac14\sum_{k=1}^\infty\frac{H_k^{(2)}}{k^2}\\
&=\frac{15}8\zeta(4)-\frac74\sum_{k=1}^\infty\frac{H_k^{(2)}}{k^2}-2\sum_{k=1}^\infty\frac{(-1)^kH_k^{(2)}}{k^2}
\end{align}
By plugging $\displaystyle\sum_{n=1}^{\infty}\frac{(-1)^nH_n^{(2)}}{n^2}=-4\operatorname{Li}_4\left(\frac12\right)+\frac{51}{16}\zeta(4)-\frac72\ln2\zeta(3)+\ln^22\zeta(2)-\frac16\ln^42\ $
( proved here ) and $\ \displaystyle\sum_{k=1}^\infty\frac{H_k^{(2)}}{k^2}=\frac74\zeta(4)\ $, we get the closed form of $\ S$
Best Answer
We can use the following fourier series:$$\frac{1}{a+b\cos t}=\frac{1}{\sqrt{a^2-b^2}}+\frac{2}{\sqrt{a^2-b^2}}\sum_{n=1}^{\infty}\left(\frac{\sqrt{a^2-b^2}-a}{b}\right)^n\cos{(nt)},\ a>b$$ Plugging $a=5, b=3$ and $t=2x$ we get: $$\frac{1}{1+3\cos^2 x}=\frac{2}{5+3\cos(2x)}=\frac{1}{2}+\sum_{n=1}^\infty (-1)^n\left(\frac{1}{3}\right)^n\cos(2nx)$$ $$\Rightarrow \int_0^\frac{\pi}{2}\frac{x^2}{1+3\cos^2 x}dx=\frac12\int_0^\frac{\pi}{2} x^2dx+\sum_{n=1}^\infty(-1)^n \left(\frac13\right)^n\int_0^\frac{\pi}{2}x^2 \cos(2nx)dx$$ $$=\frac{\pi^3}{48}+\frac{\pi}4\sum_{n=1}^\infty \left(\frac13\right)^n\frac{1}{n^2}=\frac{\pi^3}{48}+\frac{\pi}{4}\operatorname{Li}_2\left(\frac13\right)$$
Using the series obtain above, we can also conclude that: $$\sum_{n=1}^{\infty}(-1)^n \cos^n(x) \cos(nx)=-\frac13+\frac23\sum_{n=1}^\infty \left(-\frac{1}{3}\right)^n\cos(2nx)$$