With some help from Mathematica I got this result:
$$\int_0^\infty\left(5\,x^5+x\right)\operatorname{erfc}\left(x^5+x\right)dx=J_{\frac25}\left(\frac8{25\sqrt5}\right)\left(\frac8{375}\sqrt{\frac25\left(5-\sqrt5\right)}\,\pi\,J_{\frac45}\left(\frac8{25\sqrt5}\right)-\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac65}\left(\frac8{25\sqrt5}\right)\right)+J_{-\frac25}\left(\frac8{25\sqrt5}\right)\left(\frac1{75}\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{-\frac15}\left(\frac8{25\sqrt5}\right)+\frac1{25}\left(\sqrt5-5\right)\sqrt{\frac1{10}\left(5+\sqrt5\right)}\,\pi\,J_{\frac15}\left(\frac8{25\sqrt5}\right)-\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac45}\left(\frac8{25\sqrt5}\right)-\frac{4\left(\sqrt5-5\right)\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{\frac65}\left(\frac8{25\sqrt5}\right)}{1875}\right)+J_{-\frac15}\left(\frac8{25\sqrt5}\right)\left(\frac1{25}\sqrt{2\left(5-\sqrt5\right)}\,\pi\,J_{\frac25}\left(\frac8{25\sqrt5}\right)+\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac35}\left(\frac8{25\sqrt5}\right)-\frac8{375}\sqrt{\frac25\left(5-\sqrt5\right)}\,\pi\,J_{\frac75}\left(\frac8{25\sqrt5}\right)\right)+J_{\frac15}\left(\frac8{25\sqrt5}\right)\left(-\frac1{75}\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{\frac25}\left(\frac8{25\sqrt5}\right)+\frac{4\left(\sqrt5-5\right)\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{\frac35}\left(\frac8{25\sqrt5}\right)}{1875}+\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac75}\left(\frac8{25\sqrt5}\right)\right),$$
where $J_\nu(x)$ is the Bessel function of the first kind.
A solution outline:
Note that $(5\,x^5+x)=x\frac{d}{dx}(x^5+x)$. Change the integration variable $y=x^5+x$, then the integral takes the form
$$\int_0^\infty\mathcal{BR}(y)\cdot\operatorname{erfc}y\,dy,$$
where $y\mapsto\mathcal{BR}(y)$ is the inverse (properly selected to satisfy $\mathcal{BR}(y)>0$ for $y>0$) of the polynomial function $x \mapsto x^5+x$. This is a well-known non-elementary function called Bring radical, it can be used to express solutions of quintic equations in an explicit form.
An important fact, it has a representation via a generalized hypergeometric function (I used it to answer another question awhile ago). If we plug the hypergeometric representation into the integral and feed it to Mathematica, it produces the result in terms of Bessel functions shown above. I leave this step without a rigorous proof and rely on Mathematica here. The result agrees with a numerical integration to a very high precision. I would be very glad if anybody could write down an explicit derivation of the formula.
We have
$$ \int_{-\infty}^{\infty} \operatorname{erfc}^{2}(|x|) e^{-i\xi x} \, dx = \frac{4}{\xi}e^{-\xi^{2}/4} \left\{ \operatorname{erfi}\left( \frac{\xi}{2} \right) - \operatorname{erfi}\left( \frac{\xi}{2\sqrt{2}} \right) \right\}, \tag{1} $$
where $\operatorname{erfi}$ is the imaginary error function defined by
$$\operatorname{erfi}(x) = \frac{2}{\sqrt{\pi}} \int_{0}^{x} e^{t^{2}} \, dt. $$
So here goes my solution. Notice that we can write
$$ \operatorname{erfc}(|t|) = \frac{2}{\sqrt{\pi}} \int_{1}^{\infty} \left| t \right| e^{-t^{2}x^{2}} \, dx. $$
Using this, we can write
\begin{align*}
\int_{-\infty}^{\infty} \operatorname{erfc}^{2}(|t|) e^{-i\xi t} \, dt
&= \frac{4}{\pi} \int_{-\infty}^{\infty} \int_{1}^{\infty} \int_{1}^{\infty} t^{2} e^{-(x^{2}+y^{2})t^{2}} e^{-i\xi t} \, dxdydt \\
{\scriptsize(\because \text{ Fubini})}
&= \frac{4}{\pi} \int_{1}^{\infty} \int_{1}^{\infty} \left( \int_{-\infty}^{\infty} t^{2} e^{-r^{2}t^{2}} e^{-i\xi t} \, dt \right) \, dxdy. \tag{2}
\end{align*}
Using some standard complex analysis technique, we can show that
$$ \int_{-\infty}^{\infty} t^{2} e^{-r^{2}t^{2}} e^{-i\xi t} \, dt = \frac{\sqrt{\pi}}{4} \frac{2r^{2} - \xi^{2}}{r^{5}} e^{-\xi^{2} / 4r^{2}}. \tag{3} $$
Indeed, we have
\begin{align*}
\int_{-\infty}^{\infty} t^{2} e^{-r^{2}t^{2}} e^{-i\xi t} \, dt
&= e^{-\xi^{2}/4r^{2}} \int_{-\infty}^{\infty} t^{2} \exp \left\{ - r^{2} \left( t + \frac{i\xi}{2r^{2}} \right)^{2} \right\} \, dt \\
{\scriptsize(\because \text{ contour shift})}
&= e^{-\xi^{2}/4r^{2}} \int_{-\infty}^{\infty} \left( t - \frac{i\xi}{2r^{2}} \right)^{2} e^{-r^{2}t^{2}} \, dt \\
&= e^{-\xi^{2}/4r^{2}} \int_{0}^{\infty} \frac{4r^{4}t^{2} - \xi^{2}}{2r^{4}} e^{-r^{2}t^{2}} \, dt,
\end{align*}
which immediately yields $\text{(3)}$ by exploiting the gamma function. Plugging $\text{(3)}$ back to our calculation $\text{(2)}$, we get
\begin{align*}
\int_{-\infty}^{\infty} \operatorname{erfc}^{2}(|t|) e^{-i\xi t} \, dt
&= \frac{1}{\sqrt{\pi}} \int_{1}^{\infty} \int_{1}^{\infty} \frac{2r^{2} - \xi^{2}}{r^{5}} e^{-\xi^{2} / 4r^{2}} \, dxdy \\
{\scriptsize(\because \text{ symmetry})}
&= \frac{2}{\sqrt{\pi}} \iint_{1\leq y\leq x} \frac{2r^{2} - \xi^{2}}{r^{5}} e^{-\xi^{2} / 4r^{2}} \, dxdy \\
{\scriptsize(\because \text{ polar coordinate})}
&= \frac{2}{\sqrt{\pi}} \int_{0}^{\frac{\pi}{4}} \int_{\csc\theta}^{\infty} \frac{2r^{2} - \xi^{2}}{r^{4}} e^{-\xi^{2} / 4r^{2}} \, drd\theta.
\end{align*}
Using the substitution $u = \xi / 2r$, we get
\begin{align*}
\int_{-\infty}^{\infty} \operatorname{erfc}^{2}(|t|) e^{-i\xi t} \, dt
&= \frac{8}{\xi\sqrt{\pi}} \int_{0}^{\frac{\pi}{4}} \int_{0}^{\frac{\xi}{2}\sin\theta} (1 - 2u^{2}) e^{-u^{2}} \, dud\theta \\
&= \frac{8}{\xi\sqrt{\pi}} \int_{0}^{\frac{\pi}{4}} \left[ u e^{-u^{2}} \right]_{0}^{\frac{\xi}{2}\sin\theta} \, d\theta \\
&= \frac{4}{\sqrt{\pi}} \int_{0}^{\frac{\pi}{4}} \sin \theta \exp\left( -\frac{\xi^{2}}{4} \sin^{2}\theta \right) \, d\theta.
\end{align*}
Finally, using the substitution $v = \frac{1}{2}\xi \cos\theta$, we get
\begin{align*}
\int_{-\infty}^{\infty} \operatorname{erfc}^{2}(|t|) e^{-i\xi t} \, dt
&= \frac{8}{\xi \sqrt{\pi}} e^{-\frac{1}{4}\xi^{2}} \int_{\xi/2\sqrt{2}}^{\xi/2} e^{v^{2}} \, dv
= \frac{4}{\xi} e^{-\frac{1}{4}\xi^{2}} \left[ \operatorname{erfi}(v) \right]_{\xi/2\sqrt{2}}^{\xi/2}.
\end{align*}
This proves $\text{(1)}$ as desired.
Best Answer
If $g(x)$ is periodic with a period $T$, then $$\int_{k}^{T+k} g(x) dx=\int_{0}^{T} g(x) dx ~~~~~(1)$$ $$I=\int_{0}^{2\pi} \text{Erfc}[\cos(a+t)] dt=\int_{a}^{2\pi+a} \text{Erfc}[\cos x] dx=\int_{0}^{2\pi} \text{Erfc}(\cos x) dx~~~~(2)$$ Next note the property that $$\int_{0}^{2a} f(x) dx=\int_{0}^{a}[ f(x)x+ f(2a-x)] dx~~~~(3).$$ As $\cos(2\pi-x)=\cos x,$ we get $$I=2\int_{0}^{\pi} \text{Erf}(\cos x) dx~~~~(4)$$ Again using (3), we get $$I=2[\int_{0}^{\pi/2}[\text{Erfc}(\cos x)+ \text{Erfc} (-\cos x)]dx=2\pi,~~~~(5)$$ as $\text{Erfc}(z)+\text{Erfc}(-z)=2$.
Edit: Now we take up the pther integral $$J=\int_{0}^{2\pi} [\text{Erfc}(a+\sin x) ~\text{Erfc}(a+\cos x)] dx$$ Due to the property (1) again, we get $J$ independent of $a$ $$J=\int_{0}^{2\pi} [\text{Erfc}(\sin x) ~\text{Erfc}(\cos x)] dx$$ Using (3) again, we get $$J=\int_{0}^{\pi} [[\text{Erfc}(\sin x) ~\text{Erfc}(\cos x)+[\text{Erfc}(-\sin x) ~\text{Erfc}(\cos x)] dx$$ Next, use $\text{Erf}(-z)=2-\text{Erf}(z)$, to write from (4) and (5) $$J=2\int_{0}^{\pi} \text{Erf}(\cos x) dx= I=2\pi$$