The statement of the dominated convergence theorem (DCT) is as follows:
"Sequential" DCT. Suppose $\{f_n\}_{n=1}^\infty$ is a sequence of (measurable) functions such that $|f_n| \le g$ for some integrable function $g$ and all $n$, and $\lim_{n\to\infty}f_n = f$ pointwise almost everywhere. Then, $f$ is an integrable function and $\int |f-f_n| \to 0$. In particular, $\lim_{n\to\infty}\int f_n = \int f$ (by the triangle inequality). This can be written as
$$ \lim_{n\to\infty}\int f_n = \int \lim_{n\to\infty} f_n.$$
(The statement and conclusion of the monotone convergence theorem are similar, but it has a somewhat different set of hypotheses.)
As you note, the statements of these theorems involve sequences of functions, i.e., a $1$-discrete-parameter family of functions $\{f_n\}_{n=1}^\infty$. To apply these theorems to a $1$-continuous-parameter family of functions, say $\{f_\epsilon\}_{0<\epsilon<\epsilon_0}$, one typically uses a characterization of limits involving a continuous parameter in terms of sequences:
Proposition. If $f$ is a function, then
$$\lim_{\epsilon\to0^+}f(\epsilon) = L \iff \lim_{n\to\infty}f(a_n) = L\quad \text{for $\mathbf{all}$ sequences $a_n\to 0^+$.}$$
With this characterization, we can formulate a version of the dominated convergence theorem involving continuous-parameter families of functions (note that I use quotations to title these versions of the DCT because these names are not standard as far as I know):
"Continuous" DCT. Suppose $\{f_\epsilon\}_{0<\epsilon<\epsilon_0}$ is a $1$-continuous-parameter family of (measurable) functions such that $|f_\epsilon| \le g$ for some integrable function $g$ and all $0<\epsilon<\epsilon_0$, and $\lim_{\epsilon\to0^+}f_\epsilon=f$ pointwise almost everywhere. Then, $f$ is an integrable function and $\int |f-f_\epsilon|\to 0$ as $\epsilon\to 0^+$. In particular,
$$ \lim_{\epsilon\to0^+}\int f_\epsilon = \int \lim_{\epsilon\to0^+} f_\epsilon.$$
The way we use the continuous DCT in practice is by picking an arbitrary sequence $\pmb{a_n\to 0^+}$ and showing that the hypotheses of the "sequential" DCT are satisfied for this arbitrary sequence $a_n$, using only the assumption that $a_n\to 0^+$ and properties of the family $\{f_\epsilon\}$ that are known to us.
First, expand $1/(1-x)$ into its geometric series to get
$$-\int_0^1\sum_{k\ge 0} x^mx^p\log x\,dx$$
Now, consider the partial sums
$$S_N=\sum_{k=0}^Nx^k=\frac{1-x^{N+1}}{1-x}$$
We can easily show that $S_N\le S_{N+1}$, as $S_{N+1}-S_N=x^{N+1}\ge 0$ as $x\in[0,1]$.
By the monotone convergence theorem,
\begin{align}
-\int_0^1\sum_{k\ge 0} x^k x^p\log x\,dx&=-\int_0^1\lim_{N\to\infty}\sum_{k= 0}^N x^k x^p\log x\,dx \\
&=-\lim_{N\to\infty}\int_0^1\sum_{k= 0}^N x^k x^p\log x\,dx \tag{1} \\
&=-\lim_{N\to\infty}\sum_{k= 0}^N\int_0^1 x^k x^p\log x\,dx \\
&=-\sum_{k\ge 0}\int_0^1 x^k x^p\log x\,dx \\
&=\sum_{k\ge 0}\frac{1}{(k+p+1)^2} \tag{2}\\
&=\sum_{k\ge 1}\frac{1}{(k+p)^2} \\
\end{align}
Where the monotone convergence theorem was used in $(1)$ and integration by parts in $(2)$.
Best Answer
I don't think any of the standard theorems directly apply here. We can, however, still make use of the dominated convergence theorem after splitting the integral at an appropriate point (similar to your own answer).
Define $f,g,h \colon (2,\infty) \times \mathbb{R} \to \mathbb{R}$ by ($1_I$ is the indicator function of the set $I$) \begin{align} f(R,x) &= \frac{1_{[1,R]}(x)}{\sqrt{1-x^2/R^2}} \left(\frac{1}{\sqrt{x^2-1}} - \frac{1}{x}\right) \, , \\ g(R,x) &= \frac{1_{[1,R/2]}(x)}{\sqrt{1-x^2/R^2}} \left(\frac{1}{\sqrt{x^2-1}} - \frac{1}{x}\right) \, , \\ h(R,x) &= \frac{1_{(R/2,R]}(x)}{\sqrt{1-x^2/R^2}} \left(\frac{1}{\sqrt{x^2-1}} - \frac{1}{x}\right) \, . \end{align} Then $f = g + h$. For $R > 2$ and $x \in \mathbb{R}$ we have $$ g(R,x) \leq \frac{2}{\sqrt{3}} \left(\frac{1}{\sqrt{x^2-1}} - \frac{1}{x}\right) 1_{[1,\infty)} (x) $$ (the right-hand side is integrable on $\mathbb{R}$), $$ \lim_{R \to \infty} g(R,x) = \left(\frac{1}{\sqrt{x^2-1}} - \frac{1}{x}\right) 1_{[1,\infty)} (x) $$ and $$ h(R,x) = \frac{1_{(R/2,R]}(x)}{\sqrt{1-x^2/R^2}x \sqrt{x^2-1} (x + \sqrt{x^2-1})} \leq \frac{4 \cdot 1_{[0,R]}(x)}{\sqrt{1-x^2/R^2} (R^2 - 4)^{3/2}} \, . $$ Therefore, we can apply the dominated convergence theorem to the integral over $g$, while the integral over $h$ goes to zero: $$ \int \limits_{\mathbb{R}} h(R,x) \, \mathrm{d} x \leq \frac{4}{(R^2-4)^{3/2}} \int \limits_0^R \frac{\mathrm{d} x}{\sqrt{1-x^2/R^2}} = \frac{2\pi R}{(R^2-4)^{3/2}} \overset{R \to \infty}{\longrightarrow} 0 \, . $$ This yields $$ \lim_{R \to \infty} \int \limits_{\mathbb{R}} f(R,x) \, \mathrm{d} x = \lim_{R \to \infty} \int \limits_{\mathbb{R}} g(R,x) \, \mathrm{d} x \overset{\text{DCT}}{=} \int \limits_1^\infty \left(\frac{1}{\sqrt{x^2-1}} - \frac{1}{x}\right) \, \mathrm{d} x = \log(2) $$ as claimed.