The third question Q3 is basically the subject of a very recent work by N.P. Landsman.
In quantum theory, spontaneous symmetry breaking requires the system to be infinite dimensional. When the number of degrees of freedom is finite, spontaneous symmetry breaking does not take place. Consider for example a particle in one dimension moving in the potential of a double well, tunneling takes place between the two degenerate states, corresponding to the minima of the potential, resulting a unique linear superposition ground state. In the infinite number of degrees of freedom limit. The transition probabilities between the degenerate states vanish, thus partitioning the Hilbert space into mutually inaccessible sectors built up over each ground state.
It is well known that in classical systems with finite numbers of
degrees of freedom spontaneous symmetry breaking is possible as
emphasized also in the following review by Narnhofer and Thirring. Given that a (pure) state in a classical system is a point in phase space (a mixed state is a probability distribution over the phase space); then classical spontaneous symmetry breaking means that there are initial conditions leading to time invariant solutions which are not invariant under the symmetry group. For example, in the double well placing the particle in one well without enough energy to get out describes a spontaneously broken state.
There are many other examples of finite classical systems exhibiting
spontaneous symmetry breaking, the most known one is, may be, the
buckling of bars, another example is the Bead, Hoop, and Spring system .
Now, as Landsman emphasizes, Large $N$ quantum systems are analogous to
classical systems in the sense that quantum correlations vanish as
$\frac{1}{N}$, which leads to the posed question that for a finite $N$
no matter how large it is, spontaneous symmetry breaking is not allowed
whereas in the thermodynamic limit the system becomes infinite and the
spontaneous symmetry breaking is allowed. The same question can be asked for $\hbar \rightarrow 0$.
Landsman explanation is that when N becomes very large, the system
becomes exponentially unstable to a symmetry breaking perturbation which drives the system to one of the degenerate states already at a very large but finite $N$. Landsman performs the analysis by means of algebraic quantum mechanics and a full comprehension of the article needs acquaintance with his previous work.
1.
The key point that you're missing is that spontaneous symmetry breaking, or indeed the notion of phase transitions in general, only works for systems with local interactions. A phase transition is defined to be a point in Hamiltonian parameter space at which the free-energy density becomes non-analytic in the infinite-size limit. This definition obviously presupposes the existence of a well-defined infinite-system limit of the free energy density. But for translationally invariant lattice systems like the Ising model, the free energy density only approaches a constant value as $N \to \infty$ if $\sum_j J_{ij}$ converges absolutely, which roughly means that $|J_{ij}|$ has to fall off faster than $1/r^d$, where $d$ is the number of dimensions. In other words, the couplings must be reasonably local.
(Experts might object that disordered systems with nonlocal all-to-all couplings, like the Sherrington-Kirkpatrick or SYK models, still have replica-symmetry-breaking phase transitions. But that's actually only true if you rescale the coupling constants as a power of the total system size, which is not a very physical thing to do. If you don't do this, then the phase transition goes away, and indeed the $N \to \infty$ limit becomes ill-defined. Real systems are never truly all-to-all coupled - in practice there's some maximum distance at which the couplings go away, and all-to-all-coupled models are just a convenient approximation.)
Any putative explanation of spontaneous symmetry breaking that doesn't explicitly use locality is at best seriously incomplete. Decoherence is too complicated to explain here, but a key assumption is that the interactions are local in space, which picks out the position basis as a naturally favored pointer basis, so that position near-eigenstates are more natural than, say, momentum near-eigenstates.
- and 3.
The locality of the system, and specifically the assumption that perturbations are all local, gives us a notion of the "distance" between two states which is more useful than mere othogonality. As you point out, orthogonality/inner products alone can't distingush between two states that only differ by a single spin, and two states that differ by all their spins, even though the latter pair is clearly in some sense "more different" than the former.
You are of course correct that $\langle i | A | j \rangle = 0$ for any two distinct eigenstates of any Hermitian operator, not just the Hamiltonian. But that simple matrix element isn't the right definition of "the tunneling amplitude". As far as I know, the actual definition is a little fuzzy and the concept is more of an art than a science, but here are two possible conceptualizaions:
a) You can think of the symmetry-breaking term as a perturbation and decompose the Hamiltonian as $H = H_0 + \Delta H$, where $H_0$ respects the symmetry and $\Delta H$ breaks it. Then perturbation theory tells us that all the perturbative corrections can be expressed in terms of the matrix elements $\langle i_0 | \Delta H | j_0 \rangle$ where $\langle i_0|$ and $|j_0\rangle$ are the eigenstates of the unperturbed Hamiltonian $H_0$, not the exact Hamiltonian. These matrix elements are generically nonzero.
b) I don't like perturbation theory, so I prefer to think of it by analogy with Monte Carlo. The environment is constantly try to act on the system with random little local symmetry-breaking perturbations. You can think of it as if $h = 0$ in the full Hamiltonian, but $h_i \sigma_i^x$ terms randomly appear momentarily at individual sites $i$ (or similar terms on small local clusters of sites). These are like Monte Carlo candidate spin flips, and at low temperature they usually only get accepted if they lower the system's total energy. For a small system that starts in the all-$\uparrow$ state, you might get lucky and accept enough flips to eventually take you into a majority-$\downarrow$ state, at which point you'll then probably proceed to all-$\downarrow$ - even though each of those first few individual flips were unlikely. But in order to flip more than half the system, you initially need to get lucky many (independent) times in a row, and the odds of that happening decrease exponentially with system size. The "tunneling amplitude" is basically the probability of this happening after many Monte Carlo sweeps, and it indeed decreases exponentially with system size. For a small system, you'll eventually flip over to the other ground state, although it'll take a really long time. For a large system, it'll take a really long time, and for an infinite system it'll never fully get there.
If that analogy's too classical for your taste, you can instead think of the space of random quantum circuits, with circuits weighted according to a cost function that depends on the Hamiltonian matrix elements, and the "tunneling amplitude" between two quantum states is like the total weight of all the random circuits that take one state to the other.
4.
You're right that any finite value of $h$ breaks the symmetry. For any system, even an infinite one, you have that $m(h) \neq 0$ if $h > 0$. But what about the limit $h \to 0^+$? One definition of SSB is the failure of the limits $h \to 0^+$ and $N \to \infty$ to commute. In the SSB phase, after you take $N \to \infty$, you have that $m(h)$ has a jump discontinuity at $h = 0$, so that $m(0) = 0$ but $\lim \limits_{h \to 0^+} m(h) > 0$. That what we mean when we say that an infinitesimal perturbation $h$ breaks the symmetry.
Best Answer
The question is based on misinterpretation of the meaning of terms symmetry and spontaneous symmetry breaking.
Symmetry
The Hamiltonian in the OP possesses a rotational symmetry (note that all this is easily generalizable to N sites with periodic boundary conditions). This does not mean that the eigenfunstions functions should possess the same symmetry, but only that in symmetry transformations they transform into linear combinations of the eigenfunctions, which they surely do. Where the symmetry does manifest itself is that the eigenstates correspond to the irreducible representations of the symmetry group, which they do.
Spontaneous symmetry breaking
Spontaneous symmetry breaking applies to many-body systems with symmetric Hamiltonians exhibiting non-symmetric states, such as the well-defined polarization of a ferromagnet. This is essentially a many-body/statistical-physics phenomenon, which cannot be uncovered in a system with a finite number of degrees of freedom (i.e., as @PaulMalinowski pointed in their answer, it is possible only in in thermodynamic limit).
The standard basic reading here is the article by Phil Anderson More is Different. It could be particularly relevant here, since Anderson considers an example of an ammonia molecule, which is somewhat similar to the one in the OP.