It is very important to distinguish whether the symmetry is broken explicitly or spontaneously. I think that the sentence "Now when I break this symmetry spontaneously (or explicitly)" indicates that its author isn't quite distinguishing these things.
An explicit symmetry breaking generally lifts the degeneracy because the different parts of the multiplets no longer have the same energy.
However, spontaneous symmetry breaking increases the degeneracy, particularly of the ground state. It's really how spontaneous symmmetry is defined. It's a fate of the symmetry that remains the symmetry of the laws of physics, but in practice, the environment we encounter, starting from the ground state, is no longer invariant under the symmetry.
Its not being invariant means nothing else than the fact that if we act with a generator $G$ of the continuous symmetry on the ground state, we get
$$ G|0\rangle \neq 0 $$
We would get zero if the symmetry were not spontaneously broken. If it is broken, we get a nonzero vector that is independent from the original $|0\rangle$, so we get another "copy" of the ground state. Here, $G$ still commutes with the Hamiltonian $H$ so these copies have the same energy – we have degeneracy. For example, if the electroweak symmetry were global, just to simplify things, the ground state of the Higgs field could have vev $(0,246)$ in the units of GeV, but it could have any other vev with the same magnitude, too. So there are infinitely many vacua. (In gauge theory, they're made equivalent, but if we spontaneously break a global symmetry, they are states related by the symmetry and therefore having the same energy, but distinct elements of the Hilbert space.)
So the claim that the OP is dissatisfied with really holds completely generally, by definition of the spontaneous symmetry breaking! Misunderstanding that the ground state of a spontaneously broken symmetric theory is degenerate is the misunderstanding of the basic idea of the spontaneous symmetry breaking.
Note that spontaneous symmetry breaking still effectively means that the "symmetry is broken for most practical purposes" because the "copies" mentioned above may be imagined to be physically identified and we split the Hilbert space into "superselection sectors" each of which is built upon one copy of the ground state. The action of the symmetry generator on any excited state gives us a vector from another superselection sector which can't be identified with zero so from the perspective of a single superselection sector, the symmetry is simply broken.
None of these questions and discussions about explicit vs spontaneous symmetry breaking has anything to do with the time-reversal symmetry (or the absence of it) which is just another symmetry (one given by an antiunitary transformation, so some of the comments above wouldn't apply to this symmetry). Both systems with and without time-reversal symmetry satisfy the claim that the ground state is degenerate if a symmetry is spontaneously broken.
1.
The key point that you're missing is that spontaneous symmetry breaking, or indeed the notion of phase transitions in general, only works for systems with local interactions. A phase transition is defined to be a point in Hamiltonian parameter space at which the free-energy density becomes non-analytic in the infinite-size limit. This definition obviously presupposes the existence of a well-defined infinite-system limit of the free energy density. But for translationally invariant lattice systems like the Ising model, the free energy density only approaches a constant value as $N \to \infty$ if $\sum_j J_{ij}$ converges absolutely, which roughly means that $|J_{ij}|$ has to fall off faster than $1/r^d$, where $d$ is the number of dimensions. In other words, the couplings must be reasonably local.
(Experts might object that disordered systems with nonlocal all-to-all couplings, like the Sherrington-Kirkpatrick or SYK models, still have replica-symmetry-breaking phase transitions. But that's actually only true if you rescale the coupling constants as a power of the total system size, which is not a very physical thing to do. If you don't do this, then the phase transition goes away, and indeed the $N \to \infty$ limit becomes ill-defined. Real systems are never truly all-to-all coupled - in practice there's some maximum distance at which the couplings go away, and all-to-all-coupled models are just a convenient approximation.)
Any putative explanation of spontaneous symmetry breaking that doesn't explicitly use locality is at best seriously incomplete. Decoherence is too complicated to explain here, but a key assumption is that the interactions are local in space, which picks out the position basis as a naturally favored pointer basis, so that position near-eigenstates are more natural than, say, momentum near-eigenstates.
- and 3.
The locality of the system, and specifically the assumption that perturbations are all local, gives us a notion of the "distance" between two states which is more useful than mere othogonality. As you point out, orthogonality/inner products alone can't distingush between two states that only differ by a single spin, and two states that differ by all their spins, even though the latter pair is clearly in some sense "more different" than the former.
You are of course correct that $\langle i | A | j \rangle = 0$ for any two distinct eigenstates of any Hermitian operator, not just the Hamiltonian. But that simple matrix element isn't the right definition of "the tunneling amplitude". As far as I know, the actual definition is a little fuzzy and the concept is more of an art than a science, but here are two possible conceptualizaions:
a) You can think of the symmetry-breaking term as a perturbation and decompose the Hamiltonian as $H = H_0 + \Delta H$, where $H_0$ respects the symmetry and $\Delta H$ breaks it. Then perturbation theory tells us that all the perturbative corrections can be expressed in terms of the matrix elements $\langle i_0 | \Delta H | j_0 \rangle$ where $\langle i_0|$ and $|j_0\rangle$ are the eigenstates of the unperturbed Hamiltonian $H_0$, not the exact Hamiltonian. These matrix elements are generically nonzero.
b) I don't like perturbation theory, so I prefer to think of it by analogy with Monte Carlo. The environment is constantly try to act on the system with random little local symmetry-breaking perturbations. You can think of it as if $h = 0$ in the full Hamiltonian, but $h_i \sigma_i^x$ terms randomly appear momentarily at individual sites $i$ (or similar terms on small local clusters of sites). These are like Monte Carlo candidate spin flips, and at low temperature they usually only get accepted if they lower the system's total energy. For a small system that starts in the all-$\uparrow$ state, you might get lucky and accept enough flips to eventually take you into a majority-$\downarrow$ state, at which point you'll then probably proceed to all-$\downarrow$ - even though each of those first few individual flips were unlikely. But in order to flip more than half the system, you initially need to get lucky many (independent) times in a row, and the odds of that happening decrease exponentially with system size. The "tunneling amplitude" is basically the probability of this happening after many Monte Carlo sweeps, and it indeed decreases exponentially with system size. For a small system, you'll eventually flip over to the other ground state, although it'll take a really long time. For a large system, it'll take a really long time, and for an infinite system it'll never fully get there.
If that analogy's too classical for your taste, you can instead think of the space of random quantum circuits, with circuits weighted according to a cost function that depends on the Hamiltonian matrix elements, and the "tunneling amplitude" between two quantum states is like the total weight of all the random circuits that take one state to the other.
4.
You're right that any finite value of $h$ breaks the symmetry. For any system, even an infinite one, you have that $m(h) \neq 0$ if $h > 0$. But what about the limit $h \to 0^+$? One definition of SSB is the failure of the limits $h \to 0^+$ and $N \to \infty$ to commute. In the SSB phase, after you take $N \to \infty$, you have that $m(h)$ has a jump discontinuity at $h = 0$, so that $m(0) = 0$ but $\lim \limits_{h \to 0^+} m(h) > 0$. That what we mean when we say that an infinitesimal perturbation $h$ breaks the symmetry.
Best Answer
The third question Q3 is basically the subject of a very recent work by N.P. Landsman.
In quantum theory, spontaneous symmetry breaking requires the system to be infinite dimensional. When the number of degrees of freedom is finite, spontaneous symmetry breaking does not take place. Consider for example a particle in one dimension moving in the potential of a double well, tunneling takes place between the two degenerate states, corresponding to the minima of the potential, resulting a unique linear superposition ground state. In the infinite number of degrees of freedom limit. The transition probabilities between the degenerate states vanish, thus partitioning the Hilbert space into mutually inaccessible sectors built up over each ground state.
It is well known that in classical systems with finite numbers of degrees of freedom spontaneous symmetry breaking is possible as emphasized also in the following review by Narnhofer and Thirring. Given that a (pure) state in a classical system is a point in phase space (a mixed state is a probability distribution over the phase space); then classical spontaneous symmetry breaking means that there are initial conditions leading to time invariant solutions which are not invariant under the symmetry group. For example, in the double well placing the particle in one well without enough energy to get out describes a spontaneously broken state.
There are many other examples of finite classical systems exhibiting spontaneous symmetry breaking, the most known one is, may be, the buckling of bars, another example is the Bead, Hoop, and Spring system .
Now, as Landsman emphasizes, Large $N$ quantum systems are analogous to classical systems in the sense that quantum correlations vanish as $\frac{1}{N}$, which leads to the posed question that for a finite $N$ no matter how large it is, spontaneous symmetry breaking is not allowed whereas in the thermodynamic limit the system becomes infinite and the spontaneous symmetry breaking is allowed. The same question can be asked for $\hbar \rightarrow 0$.
Landsman explanation is that when N becomes very large, the system becomes exponentially unstable to a symmetry breaking perturbation which drives the system to one of the degenerate states already at a very large but finite $N$. Landsman performs the analysis by means of algebraic quantum mechanics and a full comprehension of the article needs acquaintance with his previous work.