Sometimes just using the integral form of error function can save you.
$\int_0^t\dfrac{1}{x}e^{-\frac{a^2}{x}}
\text{erf}\biggl(\dfrac{b}{\sqrt{x}}\biggr)~dx$
$=\dfrac{2}{\sqrt{\pi}}\int_0^t\dfrac{1}{x}e^{-\frac{a^2}{x}}\int_0^\frac{b}{\sqrt{x}}e^{-u^2}~du~dx$
$=\dfrac{2}{\sqrt{\pi}}\int_0^t\dfrac{1}{x}e^{-\frac{a^2}{x}}\int_0^1e^{-\left(\frac{bu}{\sqrt{x}}\right)^2}~d\biggl(\dfrac{bu}{\sqrt{x}}\biggr)~dx$
$=\dfrac{2}{\sqrt{\pi}}\int_0^t\dfrac{1}{x}e^{-\frac{a^2}{x}}\int_0^1\dfrac{b}{\sqrt{x}}e^{-\frac{b^2u^2}{x}}~du~dx$
$=\dfrac{2b}{\sqrt{\pi}}\int_0^t\int_0^1\dfrac{e^{-\frac{b^2u^2+a^2}{x}}}{x\sqrt{x}}du~dx$
$=\dfrac{2b}{\sqrt{\pi}}\int_0^1\int_0^t\dfrac{e^{-\frac{b^2u^2+a^2}{x}}}{x\sqrt{x}}dx~du$
$=\dfrac{4b}{\sqrt{\pi}}\int_0^1\int_0^t-e^{-\frac{b^2u^2+a^2}{x}}~d\biggl(\dfrac{1}{\sqrt{x}}\biggr)~du$
$=\dfrac{4b}{\sqrt{\pi}}\int_0^1\int_\infty^\frac{1}{\sqrt{t}}-e^{-(b^2u^2+a^2)x^2}~dx~du$
$=\dfrac{4b}{\sqrt{\pi}}\int_0^1\int_\frac{1}{\sqrt{t}}^\infty e^{-(b^2u^2+a^2)x^2}~dx~du$
$=\dfrac{4b}{\sqrt{\pi}}\int_0^1\int_0^\infty e^{-(b^2u^2+a^2)x^2}~dx~du-\dfrac{4b}{\sqrt{\pi}}\int_0^1\int_0^\frac{1}{\sqrt{t}}e^{-(b^2u^2+a^2)x^2}~dx~du$
$=2b\int_0^1\dfrac{1}{\sqrt{b^2u^2+a^2}}du-\dfrac{4b}{\sqrt{\pi}}\int_0^1\biggl[\sum\limits_{n=0}^\infty\dfrac{(-1)^n(b^2u^2+a^2)^nx^{2n+1}}{n!(2n+1)}\biggr]_0^\frac{1}{\sqrt{t}}~du$
$=2b\int_0^1\dfrac{1}{\sqrt{b^2u^2+a^2}}du-\dfrac{4b}{\sqrt{\pi}}\int_0^1\sum\limits_{n=0}^\infty\dfrac{(-1)^n(b^2u^2+a^2)^n}{t^{n+\frac{1}{2}}n!(2n+1)}du$
$=2b\int_0^1\dfrac{1}{\sqrt{b^2u^2+a^2}}du-\dfrac{4b}{\sqrt{\pi}}\int_0^1\sum\limits_{n=0}^\infty\sum\limits_{k=0}^n\dfrac{(-1)^nC_k^na^{2n-2k}b^{2k}u^{2k}}{t^{n+\frac{1}{2}}n!(2n+1)}du$
$=2b\biggl[\dfrac{\ln\left(b^2u+b\sqrt{b^2u^2+a^2}\right)}{b}\biggr]_0^1-\dfrac{4b}{\sqrt{\pi}}\biggl[\sum\limits_{n=0}^\infty\sum\limits_{k=0}^n\dfrac{(-1)^na^{2n-2k}b^{2k}u^{2k+1}}{t^{n+\frac{1}{2}}k!(n-k)!(2n+1)(2k+1)}\biggr]_0^1$
$=2\ln\left(b^2+b\sqrt{a^2+b^2}\right)-2\ln(|a|b)-\sum\limits_{n=0}^\infty\sum\limits_{k=0}^n\dfrac{(-1)^n4a^{2n-2k}b^{2k+1}}{\sqrt{\pi}t^{n+\frac{1}{2}}k!(n-k)!(2n+1)(2k+1)}$
$\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}\text{erfc}\left(\sqrt{\dfrac{a^2+x^2}{t}}\right)~dx$
$=\dfrac{2}{\sqrt{\pi}}\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}\int_\sqrt{\frac{a^2+x^2}{t}}^\infty e^{-u^2}~du~dx$
$=\dfrac{2}{\sqrt{\pi}}\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}\int_0^\infty e^{-u^2}~du~dx-\dfrac{2}{\sqrt{\pi}}\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}\int_0^\sqrt{\frac{a^2+x^2}{t}}e^{-u^2}~du~dx$
$=\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}dx-\dfrac{2}{\sqrt{\pi}}\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}\biggl[\sum\limits_{n=0}^\infty\dfrac{(-1)^nu^{2n+1}}{n!(2n+1)}\biggr]_0^\sqrt{\frac{a^2+x^2}{t}}~dx$
$=\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}dx-\dfrac{2}{\sqrt{\pi}}\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}\sum\limits_{n=0}^\infty\dfrac{(-1)^n(a^2+x^2)^{n+\frac{1}{2}}}{t^{n+\frac{1}{2}}n!(2n+1)}dx$
$=\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}dx-\dfrac{2}{\sqrt{\pi}}\int_0^b\sum\limits_{n=0}^\infty\dfrac{(-1)^n(a^2+x^2)^n}{t^{n+\frac{1}{2}}n!(2n+1)}dx$
$=\int_0^b\dfrac{1}{\sqrt{a^2+x^2}}dx-\dfrac{2}{\sqrt{\pi}}\int_0^b\sum\limits_{n=0}^\infty\sum\limits_{k=0}^n\dfrac{(-1)^nC_k^na^{2n-2k}x^{2k}}{t^{n+\frac{1}{2}}n!(2n+1)}dx$
$=\left[\ln\left(x+\sqrt{a^2+x^2}\right)\right]_0^b-\dfrac{2}{\sqrt{\pi}}\biggl[\sum\limits_{n=0}^\infty\sum\limits_{k=0}^n\dfrac{(-1)^na^{2n-2k}x^{2k+1}}{t^{n+\frac{1}{2}}k!(n-k)!(2n+1)(2k+1)}\biggr]_0^b$
$=\ln\left(b+\sqrt{a^2+b^2}\right)-\ln|a|-\sum\limits_{n=0}^\infty\sum\limits_{k=0}^n\dfrac{(-1)^n2a^{2n-2k}b^{2k+1}}{\sqrt{\pi}t^{n+\frac{1}{2}}k!(n-k)!(2n+1)(2k+1)}$
With some help from Mathematica I got this result:
$$\int_0^\infty\left(5\,x^5+x\right)\operatorname{erfc}\left(x^5+x\right)dx=J_{\frac25}\left(\frac8{25\sqrt5}\right)\left(\frac8{375}\sqrt{\frac25\left(5-\sqrt5\right)}\,\pi\,J_{\frac45}\left(\frac8{25\sqrt5}\right)-\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac65}\left(\frac8{25\sqrt5}\right)\right)+J_{-\frac25}\left(\frac8{25\sqrt5}\right)\left(\frac1{75}\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{-\frac15}\left(\frac8{25\sqrt5}\right)+\frac1{25}\left(\sqrt5-5\right)\sqrt{\frac1{10}\left(5+\sqrt5\right)}\,\pi\,J_{\frac15}\left(\frac8{25\sqrt5}\right)-\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac45}\left(\frac8{25\sqrt5}\right)-\frac{4\left(\sqrt5-5\right)\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{\frac65}\left(\frac8{25\sqrt5}\right)}{1875}\right)+J_{-\frac15}\left(\frac8{25\sqrt5}\right)\left(\frac1{25}\sqrt{2\left(5-\sqrt5\right)}\,\pi\,J_{\frac25}\left(\frac8{25\sqrt5}\right)+\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac35}\left(\frac8{25\sqrt5}\right)-\frac8{375}\sqrt{\frac25\left(5-\sqrt5\right)}\,\pi\,J_{\frac75}\left(\frac8{25\sqrt5}\right)\right)+J_{\frac15}\left(\frac8{25\sqrt5}\right)\left(-\frac1{75}\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{\frac25}\left(\frac8{25\sqrt5}\right)+\frac{4\left(\sqrt5-5\right)\sqrt{2\left(5+\sqrt5\right)}\,\pi\,J_{\frac35}\left(\frac8{25\sqrt5}\right)}{1875}+\frac8{375}\sqrt{\frac25\left(5+\sqrt5\right)}\,\pi\,J_{\frac75}\left(\frac8{25\sqrt5}\right)\right),$$
where $J_\nu(x)$ is the Bessel function of the first kind.
A solution outline:
Note that $(5\,x^5+x)=x\frac{d}{dx}(x^5+x)$. Change the integration variable $y=x^5+x$, then the integral takes the form
$$\int_0^\infty\mathcal{BR}(y)\cdot\operatorname{erfc}y\,dy,$$
where $y\mapsto\mathcal{BR}(y)$ is the inverse (properly selected to satisfy $\mathcal{BR}(y)>0$ for $y>0$) of the polynomial function $x \mapsto x^5+x$. This is a well-known non-elementary function called Bring radical, it can be used to express solutions of quintic equations in an explicit form.
An important fact, it has a representation via a generalized hypergeometric function (I used it to answer another question awhile ago). If we plug the hypergeometric representation into the integral and feed it to Mathematica, it produces the result in terms of Bessel functions shown above. I leave this step without a rigorous proof and rely on Mathematica here. The result agrees with a numerical integration to a very high precision. I would be very glad if anybody could write down an explicit derivation of the formula.
Best Answer
Feynman’s Trick
Considering the integral parametrized by $b$ $$ I(b)=\int_0^{\infty} e^{-ax^2} \sinh (b x) d x $$ with $I(0)=0$.
Differentiating w.r.t. $b$ gives $$ \begin{aligned} I^{\prime}(b) & =\int_0^{\infty} x e^{-ax^2} \cosh (b x) d x \\ & =-\frac{1}{2a} \int_0^{\infty} \cosh (b x) d\left(e^{-ax^2}\right) \\ & =-\frac{1}{2a}\left(\left[e^{-ax^2} \cosh (b x)\right]_0^{\infty}+ \int_0^{\infty} e^{-ax^2} \sinh (bx) d x\right)\\&=\frac{1}{2a}+\frac{b}{2a} I(b) \end{aligned} $$
Rearranging and solving by integrating factor gives $$ \begin{aligned} & I^{\prime}(b)-\frac{b}{2 a} I(b)=\frac{1}{2 a} \\ \Leftrightarrow \quad & {\left[e^{-\frac{b^2}{4 a}} I(b)\right]^{\prime}=\frac{1}{2 a} e^{-\frac{b^2}{4 a}} } \\ \Leftrightarrow \quad & e^{-\frac{b^2}{4 a}} I(b)=\frac{1}{2 } \sqrt{\frac{\pi}{a}} \operatorname{erf}\left(\frac{b}{2 \sqrt{a}}\right) \\ \Leftrightarrow \quad & I(b)= \frac{e^{\frac{b^2}{4 a}}}{2} \sqrt{\frac{\pi}{a}} \operatorname{erf}\left(\frac{b}{2 \sqrt{a}}\right) \end{aligned} $$