The difference of the sample means will indeed converge towards a normal distribution but with variance $2$ instead of $1$.
To show this I think using characteristic functions may be helpful in this setting and in particular the Lévy's convergence theorem.
For simpler notation let
$$
{\bf X}_n = \frac{\frac{\sum_{i=1}^nX_i}{n}-\mu_X}{\sqrt{\frac{\sigma^2_X}{n}}}
$$
and
$$
{\bf Y}_m = \frac{\frac{\sum_{j=1}^mY_j}{m}-\mu_Y}{\sqrt{\frac{\sigma^2_Y}{m}}}.
$$
By independence of $(X_i)$ and $(Y_j)$, we have ${\bf X}_n \perp \!\!\! \perp {\bf Y}_m$.
The characteristic function of ${\bf X}_n - {\bf Y}_m$ is, by definition,
\begin{align*}
\phi_{{\bf X}_n - {\bf Y}_m} (t) &= \mathbb E\left[ e^{it({\bf X}_n - {\bf Y}_m )}\right] \\
&=\mathbb E \left[ e^{it{\bf X}_n} \right ] \mathbb E \left[e^{-it {\bf Y}_m}\right] \ \ \ \text{(by independence)}.
\end{align*}
The characteristic function of a normal distribution with mean $\mu$ and variance $\sigma^2$ is $\phi_{\mu,\sigma^2}(t) = e^{it\mu}e^{-\frac{\sigma^2 t^2}{2}}$.
By convergence in distribution of ${\bf X}_n$ and ${\bf Y}_m$ towards a $\mathcal{N}(0,1)$ we have
\begin{align*}
\phi_{{\bf X}_n - {\bf Y}_m} (t) &\to \left ( e^{-\frac{t^2}{2}} \right)^2 \\
&\to e^{-t^2}.
\end{align*}
The "$\to$" above states that the convergence takes place when both $n$ and $m$ converge to $+\infty$.
Since $e^{-t^2}$ is the characteristic function of a $\mathcal{N}(0,2)$, we have ${\bf X}_n - {\bf Y}_m \rightsquigarrow \mathcal{N}(0,2)$.
For the variable
$$
S=\frac{\frac{1}{n} \sum_{i=1}^n X_i - \frac{1}{m} \sum_{j=1}^m Y_j - (\mu_X - \mu_Y)}{\sqrt{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}}}
$$
we can reexpress it as:
$$
S= \frac{\sqrt{\frac{\sigma_X^2}{n}}}{\sqrt{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}}} {\bf X}_n - \frac{\sqrt{\frac{\sigma_Y^2}{m}}}{\sqrt{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}}} {\bf Y}_m
$$
Since ${\bf X}_n$ and ${\bf Y}_m$ $\to \mathcal{N}(0,1)$ by using the argument above we have $S \to \mathcal{N}(0,\sigma^2)$ where
\begin{align*}
\sigma^2 &=\lim_{n,m \to \infty} \left\{ \operatorname{Var} \left( \frac{\sqrt{\frac{\sigma_X^2}{n}}}{\sqrt{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}}} {\bf X}_n \right ) + \operatorname{Var} \left( \frac{\sqrt{\frac{\sigma_Y^2}{m}}}{\sqrt{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}}} {\bf Y}_m \right ) \right \} \\
&=\lim_{n,m \to \infty} \left \{ \frac{\frac{\sigma_X^2}{n}}{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}} \operatorname{Var} ({\bf X}_n) + \frac{\frac{\sigma_Y^2}{m}}{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}} \operatorname{Var} ({\bf Y}_m) \right \} \\
&= \lim_{n,m \to \infty} \left \{ \frac{\frac{\sigma_X^2}{n}}{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}} + \frac{\frac{\sigma_Y^2}{m}}{\frac{\sigma_X^2}{n} + \frac{\sigma_Y^2}{m}} \right \} \\
&= 1
\end{align*}
Best Answer
This answer should be viewed as being a compilation of comments which resulted in the OP's confusion being resolved, and is not intended as being the most general or rigorous statement on the subject. Its goal is to aid interpretation.
Jeremias K's answer shows the formula for the variance of a weighted sum of random variables. In particular, taking the special case of all weights being $1/n$ for a weighted sum of n independent random variables, the variance of the weighted sum equals $1/n^2$ times the sum of the variances of the individual random variables. If the random variables are also identically distributed, then the variance of the weighted sum equals $1/n$ times the variance of an individual random variable.
As you take the average of more and more i.i.d. (Gaussian) random variables, the variance of that average will decrease, as seen from the preceding formulation, which is a result which should be quite familiar. Why does the variance of that average decrease vs. an individual random variable? Well, taking the case of 2 independent random variables, an extreme value in one variable tends not to be matched with a similar direction extreme value in the other variable, hence making the average less extreme than the individual random variables, hence the decreased variance. The magnitude of this effect increases as the number of random variables being averaged increases.