An explicit antiderivative is messy, but here's an outline for evaluating this integral by hand.
First, make a linear substitution $x = \alpha u$ for an appropriate constant $\alpha$, which transforms the integral $$\int \frac{x^2 \,dx}{a x^5 + b}$$ into some constant multiple of
$$\int \frac{u^2 \,du}{1 - u^5} .$$
This is a rational expression, so in principle we can apply partial fractions and solve, but $1 - u^5$ factors over $\Bbb Q$ into $u - 1$ and a quartic polynomial irreducible over $\Bbb Q$. Thus, to factor the denominator into a product of linear and quadratic polynomials we need to resort to irrational coefficients.
Factoring a generic real quartic over $\Bbb R$ is unpleasant, but we can take advantage of the special form of the denominator: The roots of $1 - u^5$ are precisely the $5$th roots of unity, namely $1$ and the paired complex conjugates $e^{\pm 2 \pi i / 5}$ and $e^{\pm 4 \pi i / 5}$. Thus, one real quadratic factor of $1 - u^5$ is
$$(u - e^{2 \pi i / 5}) (u - e^{-2 \pi i / 5}) = u^2 - 2 \cos \left(\frac{2 \pi}{5}\right) u + 1$$
and the other is
$$(u - e^{4 \pi i / 5}) (u - e^{-4 \pi i / 5}) = u^2 + 2 \cos \left(\frac{\pi}{5}\right) u + 1 .$$
Optionally, we can rewrite these expressions using the facts that $2 \cos \frac{\pi}{5} = \phi$, where $\phi := \frac{1}{2}(1 + \sqrt{5})$ is the Golden Ratio, and $2 \cos \frac{2 \pi}{5} = \frac{1}{\phi}$.
Applying the Method of Partial Fractions thus gives a decomposition
$$\frac{u^2}{1 - u^5} = \frac{A}{u - 1} + \frac{B u + C}{u^2 + \phi u + 1} + \frac{D u + E}{u^2 - \frac{1}{\phi} u + 1}$$
for some constants $A, B, C, D, E$, so
$$\int \frac{u^2\, du}{1 - u^5} = A \int \frac{du}{u - 1} + \int \frac{(B u + C) du}{u^2 + \phi u + 1} + \int \frac{(D u + E) du}{u^2 - \frac{1}{\phi} u + 1} .$$
- The integral $$\int \frac{du}{u - 1}$$ is elementary.
- We can rewrite the integral of the second term as a linear combination of
$$\int \frac{(2 u + \phi) du}{u^2 + \phi u + 1} \qquad \textrm{and} \qquad \int \frac{du}{u^2 + \phi u + 1} .$$
The left integral can be handled with the substitution $v = u^2 + \phi u + 1, dv = (2 u + \phi) du$, which gives $$\int \frac{(2 u + \phi) du}{u^2 + \phi u + 1} = \int \frac{dv}{v} = \log |v| + K = \log (u^2 + \phi u + 1) + K .$$ A linear substitution $w = \beta u + \gamma, dw = \beta \,du$ transforms the integral on the right into a multiple of $$\int \frac{dw}{w^2 + 1} = \arctan w + K' = \arctan (\beta u + \gamma) + K' .$$
- The third integral can be handled much like the second integral.
With all of the integrals in $u$ now expressed in terms of elementary functions, all that remains is to undo the original substitution, that is, back-substitute $u = \frac{x}{\alpha}$ to produce an antiderivative in $x$.
We can show fairly straightforwardly that this reduces to an elliptic integral, which cannot be an elementary function: put $ x = \arctan(u^2) $. Then $ dx = 2u/(1+u^4) \, du $, $\tan x = u^2$ and $\sin x = u^2/\sqrt{1+u^4}$, and rationalising implies that the integral becomes
$$ \int \bigg( 2u^4 + \frac{2u^6}{\sqrt{1+u^4}} \bigg) \, du , $$
and we just need to worry about the second term. It so happens that this was one of the earliest integrals Liouville considered when he became interested in when an integral is algebraic (See Lützen's Joseph Liouville 1809–1882 pp. 374ff. for the details). An integration by parts reduces us to $ \int \frac{u^2}{\sqrt{1+u^4}} \, du $, which is known to not be elementary (see either Liouville's work, or Ritt's book Integration in finite terms). Thus the "elementary part" is
$$ \frac{2}{5} ( u^5 + u^3 \sqrt{1+u^4}) , $$
while the non-elementary part is the elliptic integral
$$ - \frac{6}{5} \int \frac{u^2}{\sqrt{1+u^4}} \, du = \frac{6}{5}\sqrt{i} ( F(\arcsin(\sqrt{i}u) \mid -1) - F(\arcsin(\sqrt{i}u) \mid -1) . $$
One could write in terms of $x$ again, but there doesn't seem much point.
Best Answer
These integrals are often wrapped up nicely by substitutions of the form: $$u=x^a\pm\frac{1}{x^a}$$ where $a$ is chosen appropriately. A little bit of playing around leads to the following: $$\int\frac{x^{5}-x}{x^{8}+1}dx=\int\frac{x^{3}\left(x^{2}-\frac{1}{x^{2}}\right)dx}{x^{4}\left(x^{4}+\frac{1}{x^{4}}\right)}=\int\frac{\left(x^{2}-\frac{1}{x^{2}}\right)dx}{x\left[\left(x^{2}+\frac{1}{x^{2}}\right)^{2}-2\right]}$$ Now let $$u=x^{2}+\frac{1}{x^{2}}$$ $$du=2\left(x-\frac{1}{x^{3}}\right)dx=2\frac{1}{x}\left(x^{2}-\frac{1}{x^{2}}\right)dx$$ Hence $$2I=\int\frac{du}{u^{2}-2}=\frac{1}{2\sqrt{2}}\int\frac{du}{u-\sqrt{2}}-\int\frac{du}{u+\sqrt{2}}=\frac{1}{2\sqrt{2}}\ln\left|\frac{u-\sqrt{2}}{u+\sqrt{2}}\right|$$ $$I=\frac{1}{4\sqrt{2}}\ln\left|\frac{x^{4}-\sqrt{2}x^{2}+1}{x^{4}+\sqrt{2}x^{2}+1}\right|$$