Usually, integration by parts only works one way. For example, evaluating $\int xe^x\,dx$ can only be done by differentiating $x$ and integrating $e^x$, but not the other way round, since $\int x^2e^x\,dx$ makes it more difficult than the original integral.
However, the integral $$\int x\ln x\,dx$$ can be evaluated using integration by parts both ways:
$$\int x\ln x\,dx=x\int \ln x\,dx – \int\left(x'\int\ln x\,dx\right)\,dx=x^2(\ln x-1)-\int x(\ln x-1)\,dx$$ and (directly), $$\int x\ln x\,dx=\ln x\int x\,dx-\int\left(\ln'x\int x\,dx\right)\,dx=\frac{x^2}2\ln x-\int\frac x2\,dx.$$
There are also trigonometric integrals that can do this. For instance, an integral consisting of $\sin$ and $\cos$, as $\int \sin = -\cos$ and $\int \cos = \sin$: $$\int \sin2x\cos3x\,dx.$$
@JamesArathoon has provided this one as well: $$\int x\arctan x\,dx$$
What are some other examples of integrals that have this property as well?
I believe that such integrals are rather rare, so I don't want 'similar' integrals like multiplying by a constant or adding $1$ to the integrand.
Of course, integrands of forms other than $xf(x)$ would be even better (and more challenging 🙂
Best Answer
So we wish for $f(x)$ and $g(x)$ such that we can evaluate $\int f(x) g(x) dx$ using integration by parts in two ways, either by A:
$$\int f(x) g(x) dx = g(x)\int f(x) dx - \int (g'(x) \int f(x) dx ) dx $$
... or by B:
$$\int f(x) g(x) dx = f(x)\int g(x) dx - \int (f'(x) \int g(x) dx ) dx $$
In the following list, (i) to (iii) are put forward by yourself, (iv) and (v) are trivial examples from the comments, and (vi) and (vii) are new.
(i) $f(x) = x$ and $g(x) = \mathrm{ln}(x)$
(ii) $f(x) = \mathrm{sin}(mx) $ and $g(x) =\mathrm{cos}(nx) $ for $n \neq m$
(iii) $f(x) = x $ and $g(x) = \mathrm{arctan}(x) $
(iv) $f(x) = x^n $ and $g(x) = x^m $
(v) $f(x) = e^{nx} $ and $g(x) = e^{mx} $
(vi) $f(x) = e^{nx} $ and $g(x) = \mathrm{sin}(mx) $
(vii) $f(x) = x $ and $g(x) = \sqrt{1+x^2}$
Other examples can be made by switching around some trig and hyperbolic functions (e.g. $\mathrm{sin}$ and $\mathrm{cos}$ for $\mathrm{sinh}$ and $\mathrm{cosh}$ in (ii) or $\mathrm{arctan}$ for $\mathrm{arcsin}$ in (iii)).
It sometimes seems the case that one of the directions A or B is much easier than the other. I think this is true of (i), (iii) and (vii) (and I have written them so that A is easier than B).
For all of the examples except for the trivial (iv) and (v), at least one of A and B require the use of the "indirect method", where the integrand $\int f(x)g(x) dx$ is found again on the RHS and the expression is rearranged to get $\alpha \int f(x)g(x) dx$ on the LHS with $\alpha \neq 1$.
So far I cannot spot any other interesting similarities between these examples... Perhaps others can, or could add to the list?
Edit: another example
(viii) $f(x)=x$ and $g(x)=\mathrm{erf}(x)=\frac{2}{\sqrt{\pi}}\int^{x}_0 e^{-t^2} dt$
(We can show that $ \int \mathrm{erf}(x) dx = x\mathrm{erf}(x) + \frac{e^{-x^2}}{\sqrt{\pi}} + C $ by another application of integration by parts, and $\frac{d}{dx} \mathrm{erf}(x)= \frac{2}{\sqrt{\pi}} e^{-x^2} $ fairly trivially.)
Interestingly, evaluating (viii) using B requires use of the “indirect method” again.
I also tried using $f(x)=x$ and $g(x)=\Gamma (x)$, but the integral is undefined.
Edit: proof of equivalence of methods A and B when $f(x)=x$
@farruhota suggests in the comments that (i) might be based on some sort of circular argument, as $\int \mathrm{ln}(x) dx $ is often itself evaluated using integration by parts. The same could be said of $\int \mathrm{arctan}(x) dx$ in (iii) and of $\int \mathrm{erf}(x) dx$ in (viii). I now believe that this is true and is very much linked to why the "indirect method" keeps cropping up in method B. I offer the following proof.
Let $f(x)=x$. Then method A goes:
$$ \begin{align} I(x) &:= \int^x_0 tg(t) dt \\ &= \frac{1}{2}x^2 g(x)-\frac{1}{2} \int^x_0 t^2 g'(t) dt \qquad \qquad (1) \end{align} $$
... and the integral $\int^x_0 t^2 g'(t) dt$ must then be evaluated.
(Note that I am now using limits of integration in order to make my workings more rigorous.)
Method B begins instead:
$$ \begin{align} I(x) &= \int^x_0 tg(t)dt \\ &= x \int^x_0 g(t)dt - \int^x_0 \left( \int^t_0 g(s)ds \right) dt \qquad \qquad (2) \end{align} $$
Now I'll use integration by parts again in order to evaluate $\int^x_0 g(t)dt$ and $\int^t_0 g(s)ds$, so that $(2)$ follows on as:
$$ \begin{align} I(x) &= x \left( xg(x)-\int^x_0 tg'(t)dt \right) - \int^x_0 \left( tg(t) - \int^t_0 sg'(s)ds \right) dt \\ &= x^2g(x) - x \int^x_0 tg'(t)dt - I(x) + \int^x_0 \int^t_0 sg'(s)ds \, dt \end{align} $$
Now here's where the "indirect method" comes in. We can collect $I(x)$ onto the LHS and divide by $2$ so that:
$$ I(x) = \frac{1}{2} x^2g(x) - \frac{1}{2} \underbrace{ \left( x \int^x_0 tg'(t)dt - \int^x_0 \int^t_0 sg'(s)ds \, dt \right) }_{=:K(x)} \qquad \qquad (3) $$
Focus now on $K(x)$. Write $h(t):=tg'(t)$ so that:
$$ \begin{align} K(x) &= x \int^x_0 h(t)dt - \int^x_0 \int^t_0 h(s)ds \, dt \\ &= \int^x_0 th(t) dt \\ &= \int^x_0 t^2g'(t) \end{align} $$
... where I've used integration by parts "backwards" to get from line 1 to line 2.
Putting this back into $(3)$, we get:
$$ I(x)= \frac{1}{2}x^2 g(x)-\frac{1}{2} \int^x_0 t^2 g'(t) dt \qquad \qquad (4)$$
... which is exactly the same as method A at $(1)$!
Both methods A and B boil down to whether we can evaluate $\int^x_0 t^2 g'(t) dt$. I hence offer the following as a (partial) answer to the question.
If $f(x)=x$ and $g(x)$ is such that we can evaluate $\int x^2 g'(x) dx$ then $\int f(x) g(x) dx$ can be integrated by parts "both ways".
Claim: A and B are always equivalent
Now, you suggest finding examples where $f(x)$ is not just equal to $x$.
If equations $(2)$ to $(4)$ are worked through with a general $f(x)$ instead of with $f(x)=x$ then the following can be obtained:
$$ \int^x_0 f(t)g(t)dt = xf(x)g(x) - \int^x_0 tf'(t)g(t)dt - \int^x_0 tf(t)g'(t)dt \qquad (5) $$
Note that $f(x)=x$ worked nicely because $f(x)=kx \iff tf'(t)=f(t)$ and so the second term on the RHS of $(5)$ becomes $I(x)$.
Also note that $(5)$ is symmetric in $f$ and $g$. It can be obtained by beginning with either route A or B. I would hence conclude that routes A and B are always equivalent:
route A can be used iff B can be used, as both can be manipulated into $(5)$
(However, one of these routes might be much longer than the other.)