Fleshing out Ragib's idea: we have the substitution $\,t\to -u+\pi i\Longrightarrow dt=-du,$ , so we put
$$f(t):=\frac{t^2e^t}{1+e^{2t}}\Longrightarrow$$
$$\Longrightarrow\int_{R+\pi i}^{-R+\pi i}f(t)\,dt=-\int_{-R}^R\frac{(u^2-2\pi iu-\pi^2)e^{-u+\pi i}}{1+e^{-2u+2\pi i}}\,du=$$
$$=\int_{-R}^R\frac{u^2e^{-u}}{1+e^{-2u}}du-2\pi i\int_{-R}^R\frac{ue^{-u}}{1+e^{-2u}}\,du-\pi^2\int_{-R}^R\frac{e^{-u}}{1+e^{-2u}}\,du$$
The first integral above is just like the one on the real axis (BTW, note the signs in front of the two last integrals, which come from the factor $\,e^{\pi i}=-1\,$).
The last integral is
$$\pi^2\int_{-R}^R\frac{d(e^{-u})}{1+e^{-2u}}=\left.\pi^2\arctan e^{-u}\right|_{-R}^R=$$
$$=\pi^2\left(\arctan e^{-R}-\arctan e^R\right)\xrightarrow [R\to\infty]{}\pi^2\left(0-\frac{\pi}{2}\right)=-\frac{\pi^3}{2}$$
The second integral above is zero as the integrand is just $\,\frac{\pi iu}{\cosh u}\,$ , which is an odd function...
Denoting our integral by $\, I \,$, we thus have that
$$2I-\frac{\pi ^3}{2}=-\frac{\pi^3}{4}\Longrightarrow I=\frac{\pi^3}{8}$$
You should use a keyhole contour, but also consider the integral
$$\oint_C dz \frac{\log^3{z}}{z^2-z+1}$$
Evaluating this contour integral, it will turn out (proof left to reader) that the integrals along the circular arcs out to infinity and down to zero near the origin vanish. That leaves the integrals up and back along the real line:
$$\oint_C dz \frac{\log^3{z}}{z^2-z+1} = \int_0^{\infty} dx \frac{\log^3{x}}{x^2-x+1} - \int_0^{\infty} dx \frac{(\log{x}+i 2 \pi)^3}{x^2-x+1} $$
Combine the expression on the right and get
$$\oint_C dz \frac{\log^3{z}}{z^2-z+1} = i \left [-6 \pi \int_0^{\infty} dx \frac{\log^2{x}}{x^2-x+1} + 8 \pi^3 \int_0^{\infty} dx \frac{1}{x^2-x+1}\right ] \\- 4 \pi^2 \int_0^{\infty} dx \frac{\log{x}}{x^2-x+1}$$
Now set this equal to $i 2 \pi$ times the sum of the residues at the poles. The poles are at $z = e^{i \pi/3}$ and $z=e^{i 5 \pi/3}$. Note that I am not saying that the latter pole is at $z=e^{-i \pi/3}$ because that would be inconsistent with the branch I chose in defining the argument of the values just below the real line to be $2 \pi$.
The residues are, at $z = e^{i \pi/3}$:
$$\frac{-i \pi^3/27}{2 e^{i \pi/3}-1}$$
and at $z=e^{i 5 \pi/3}$:
$$\frac{-i 125 \pi^3/27}{2 e^{i 5\pi/3}-1}$$
The contour integral is then $i 2 \pi$ times the sum of these residues:
$$\oint_C dz \frac{\log^3{z}}{z^2-z+1} = i 2 \pi \frac{124 \pi^3}{27 \sqrt{3}}$$
The real part is zero, which makes sense in light of the 1st line in your post. For the imaginary part, however, we have two integrals, one of which we want, the other of which is in our way and we have to determine. Fortunately, the evaluation follows exactly the same pattern as above, for an even simpler integral:
$$\oint_C dz \frac{\log{z}}{z^2-z+1}$$
which, when analyzed as above, spawns
$$\int_0^{\infty} dx \frac{1}{x^2-x+1} = \frac{4 \pi}{3 \sqrt{3}}$$
I leave it to the reader to verify this. The problem is then a simple one in algebra and arithmetic which I also leave to the reader. I get
$$\int_0^{\infty} dx \frac{\log^2{x}}{x^2-x+1} = \frac{20 \pi^3}{81 \sqrt{3}}$$
which implies that
$$\int_0^{1} dx \frac{\log^2{x}}{x^2-x+1} = \frac{10 \pi^3}{81 \sqrt{3}}$$
ADDENDUM
I realize that I didn't fully answer the OP's question. You do not want to use semicircular contours for integrals of the form
$$\int_0^{\infty} dx\: f(x)$$
when $f$ is not even. The reason for this is that the integral over the negative real line is not the same that the integral over the positive real line. For example, let's try again to attack the integral
$$\int_0^{\infty} dx \frac{1}{x^2-x+1}$$
in this manner. This is emphatically not the same as
$$\int_{-\infty}^{\infty} dx \frac{1}{x^2-x+1}$$
which is what would result from a semicircular contour. Such problems are not alleviated by avoiding branch points at the origin. In this case, the trick of using a keyhole contour on the integral
$$\oint_C dz \: f(z) \, \log{z}$$
works best, so long as you are careful about using a consistent branch of the log as I demonstrated above. In some cases, such as
$$\int_0^{\infty} \frac{dx}{x^3+1}$$
you can get away with a wedge contour instead of a keyhole. This is because the denominator is invariant upon rotation by $2 \pi/3$.
ADDENDUM II
I just realized that I solved a very similar problem here.
Best Answer
Actually, we do need to worry about the pole at $x=1$ if we intend to use contour integration, for reasons that are a bit subtle. I will demonstrate below.
The standard way to treat integrals of rational functions times logs over $[0,\infty)$ in complex analysis is to consider a keyhole contour, and an integral over that contour of the next higher power of log. In this case, the integral is
$$\oint_C dz \frac{\log^2{z}}{z^3-1}$$
$C$, however, is a modified keyhole contour about the positive real axis of outer radius $R$ and inner radius $\epsilon$. The modification lies on small semicircular bumps above and below $z=1$ of radius $\epsilon$, and we will consider the limits as $\epsilon \to 0$ and $R\to\infty$.
Let's evaluate this integral over the contours. There are $8$ pieces to evaluate, as follows:
$$\int_{\epsilon}^{1-\epsilon} dx \frac{\log^2{x}}{x^3-1} + i \epsilon \int_{\pi}^0 d\phi \, e^{i \phi} \frac{\log^2{\left (1+\epsilon e^{i \phi}\right )}}{(1+\epsilon e^{i \phi})^3-1} \\ + \int_{1+\epsilon}^R dx \frac{\log^2{x}}{x^3-1} + i R \int_0^{2 \pi} d\theta \, e^{i \theta} \frac{\log^2{\left (R e^{i \theta}\right )}}{R^3 e^{i 3 \theta}-1} \\ + \int_R^{1+\epsilon} dx \frac{(\log{x}+i 2 \pi)^2}{x^3-1} + i \epsilon \int_{2 \pi}^{\pi} d\phi \, e^{i \phi} \frac{(\log{\left (1+\epsilon e^{i \phi}\right )}+i 2 \pi)^2}{(1+\epsilon e^{i \phi})^3-1} \\ + \int_{1-\epsilon}^{\epsilon} dx \frac{(\log{x}+i 2 \pi)^2}{x^3-1} + i \epsilon \int_{2 \pi}^0 d\phi \, e^{i \phi} \frac{\log^2{\left (\epsilon e^{i \phi}\right )}}{\epsilon^3 e^{i 3 \phi}-1} $$
(To see this, draw the contour out, including the bumps about $z=1$.)
As $R \to \infty$, the fourth integral vanishes as $\log^2{R}/R^2$. As $\epsilon \to 0$, the second integral vanishes as it is $O(\epsilon^3)$, while the eighth integral vanishes as $\epsilon \log^2{\epsilon}$. This leaves the first, third, fifth, sixth and seventh integrals, which in the above limits, become
$$PV \int_0^{\infty} dx \frac{\log^2{x} - (\log{x}+i 2 \pi)^2}{x^3-1} + i \frac{4 \pi^3}{3}$$
EDIT
It should be appreciated that, in the fifth, sixth, and seventh integrals, the $i 2 \pi $ factor appears because, on the lower branch of the real axis, we write $z=x \, e^{i 2 \pi}$. In the sixth integral, in fact, $z = e^{i 2 \pi} + \epsilon \, e^{i \phi + 2 \pi}$.
END EDIT
The $PV$ denotes the Cauchy principal value of the integral. As it stands, the integral does not actually converge. Nevertheless, we are not actually considering the integral straight through the pole at $z=1$, but a very small detour around the pole. Thus, in the limit, we get the Cauchy PV. A little rearranging cancels the $\log^2$ term, and we now have two integrals to evaluate:
$$-i 4 \pi \int_0^{\infty} dx \frac{\log{x}}{x^3-1} + 4 \pi^2 PV \int_0^{\infty} \frac{dx}{x^3-1} + i \frac{4 \pi^3}{3}$$
Note we could remove the $PV$ on the first integral because the pole is a removable singularity.
The contour integral is equal to $i 2 \pi$ times the sum of the residues at the poles. The poles here are at $z=e^{i 2 \pi/3}$ and $z=e^{i 4 \pi/3}$. Note that the pole at $z=1$ is not inside the contour $C$ because of the detour around that "pole". It should be appreciated that the poles must have their arguments between $[0,2 \pi]$ because of the way we defined $C$.
In any case, we now have that the above 1D integrals over the positive real line are equal to
$$i 2 \pi \left [\frac{-4 \pi^2/9}{3 e^{i 4 \pi/3}} + \frac{-16 \pi^2/9}{3 e^{i 8 \pi/3}} \right ] = -\frac{4 \pi ^3}{3 \sqrt{3}}+i \frac{20 \pi ^3}{27} $$
Equating real and imaginary parts, we find that
$$ \int_0^{\infty} dx \frac{\log{x}}{x^3-1} = \frac{4 \pi^2}{27} $$
$$ PV \int_0^{\infty} \frac{dx}{x^3-1} = -\frac{\pi}{3 \sqrt{3}} $$