Notice that for $a>0$, we have $$F(ax) = e^{-a^{2}x^{2}}\int_{0}^{ax} e^{y^{2}} \mathrm dy = e^{-a^{2}x^{2}} \int_{0}^{a} u e^{x^{2}u^{2}} \, \mathrm du . \tag{1}$$
Then using $(1)$, we get
$$ \begin{align} I &= \int_{0}^{\infty} F(x) F(x \sqrt{2}) \, \frac{e^{-x^{2}}}{x^{2}} \, \mathrm dx \\&= \int_{0}^{\infty} \int_{0}^{\sqrt{2}} \int_{0}^{1} x e^{-x^{2}} e^{x^{2} y^{2}} x e^{-2x^{2}} e^{x^{2}z^{2}} \, \frac{e^{-x^{2}}}{x^{2}} \, \mathrm dy \, \mathrm dz \, \mathrm dx \\ &= \int_{0}^{\sqrt{2}} \int_{0}^{1} \int_{0}^{\infty} e^{-(4-y^{2}-z^{2})x^{2}} \, \mathrm dx \, \mathrm dy \, \mathrm dz \\ &= \frac{\sqrt{\pi}}{2} \int_{0}^{\sqrt{2}} \int_{0}^{1} \frac{1}{\sqrt{4-y^{2}-z^{2}}} \, \mathrm dy \, \mathrm dz \\&= \frac{\sqrt{\pi}}{2} \int_{0}^{\sqrt{2}} \int_{0}^{\arcsin ( \frac{1}{\sqrt{4-z^{2}}})} \, \mathrm d \theta \, \mathrm d z \tag{2} \\ &= \frac{\sqrt{\pi}}{2} \int_{0}^{\sqrt{2}} \arcsin \left( \frac{1}{\sqrt{4-z^{2}}} \right) \, \mathrm dz \\ &= \frac{\sqrt{\pi}}{2} \left( \frac{ \sqrt{2} \pi}{4} - \int_{0}^{\sqrt{2}} \frac{z^{2}}{\sqrt{3-z^{2}} (4-z^{2})} \, \mathrm dz \right) \tag{3} \\ &= \frac{\sqrt{\pi}}{2} \left( \frac{\sqrt{2} \pi}{4} - \int_{1 / \sqrt{2}}^{\infty} \frac{1}{\sqrt{3u^{2}-1} (4u^{2}-1)} \frac{\mathrm du}{u}\right) \tag{4} \\ &= \frac{\sqrt{\pi}}{2} \left( \frac{\sqrt{2} \pi}{4} - 3 \int_{1 /\sqrt{2}}^{\infty} \frac{1}{ (4w^{2}+1)(w^{2}+1)} \, \mathrm dw\right) \tag{5}\\ &=\frac{\sqrt{\pi}}{2} \left( \frac{\sqrt{2} \pi}{2} - 4 \int_{1/ \sqrt{2}}^{\infty} \frac{1}{4w^{2}+1} + \int_{1/ \sqrt{2}}^{\infty} \frac{1}{w^{2}+1} \, \mathrm dw\right) \\ &= \frac{\sqrt{\pi}}{2} \left[ \frac{\sqrt{2} \pi}{4} - \pi +2 \arctan \left( \sqrt{2} \right) +\frac{\pi}{2} - \arctan \left( \frac{1}{\sqrt{2}} \right) \right] \\ &= \frac{\sqrt{\pi}}{2} \left( \frac{\sqrt{2} \pi}{4} - \pi + 3 \arctan{\sqrt{2}} \right). \end{align}$$
$(2)$ Let $y=\sqrt{4-z^{2}}\sin \theta$.
$(3)$ Integrate by parts.
$(4)$ Let $z = \frac{1}{u}$.
$(5)$ Let $w^{2}=3u^2-1$.
EDIT:
Using the same approach, I get
$$ \int_{0}^{\infty} F(ax) F(bx) \, \frac{e^{-p^{2}x^{2}}}{x^{2}} \, \mathrm dx $$
$$ = \frac{\sqrt{\pi}}{2} \left[b \arcsin \left( \frac{a}{\sqrt{a^{2}+p^{2}}} \right) - \sqrt{a^{2}+b^{2}+p^{2}} \arctan \left(\frac{ab}{p \sqrt{a^{2}+b^{2}+p^{2}}} \right) + a \arctan \left( \frac{b}{p} \right)\right]$$
where $a, b,$ and $p$ are all positive parameters.
Best Answer
Introduce $$ I(\lambda)=\int_0^\infty\mathrm{erfi}(\lambda x)\frac{\exp(-x^2)}{x}\,\mathrm{d}x $$ for $\lambda\in[0,1]$. Note that $I(\lambda)$ is continuous (and even real-analytic on $(0,1)$) and differentiating with respect to $\lambda$ under the integral sign, we have $$ I'(\lambda)=\frac{2}{\sqrt{\pi}}\int_0^\infty\exp(-(1-\lambda^2) x^2)\,\mathrm{d}x=\frac{1}{\sqrt{1-\lambda^2}} $$ for $\lambda\in (0,1)$. Furthermore, $I(0)=0$ and so $$ I=I(1)=\int_0^1\frac{1}{\sqrt{1-\lambda^2}}\,\mathrm{d}\lambda=\frac{\pi}{2} $$