Let me give it a shot:
If I interpret this correctly, $\mathbf{F}$ will be the operator for the full spin of the coupled system, $\mathbf{S}$ will be the operator of the electron spin (usually, one would consider $\mathbf{J}$, the spin containing also spin-orbit coupling, but we are on the S-shell, hence no angular momentum) and $\mathbf{I}$ will be the nuclear spin. Then it should hold that $\mathbf{F}=\mathbf{S}+\mathbf{I}$, right?
First, let's have a look at the hyperfine structure Hamiltonian $\mathbf{H}_{hf}$. By construction of $\mathbf{F}$, the eigenstates of $\mathbf{H}_{hf}$ will be eigenstates of $F^2$ and $F_z$. This is just the same as for angular momentum and electron spin (and we construct $\mathbf{F}$ to have this property - this lets us label the eigenstates by the quantum number corresponding to $\mathbf{F}$). Hence the Hamiltonian must be diagonal in the $|F^2,m_F\rangle$-basis. One can also see that $F^2$ commutes with $I^2$ and $S^2$ (and so does $F_z$), since $\mathbf{F}=\mathbf{I}+\mathbf{S}$.
Now we have a look at $\mathbf{H}_B$, the interaction Hamiltonian with a constant magnetic field. We can see that (up to some prefactor) $\mathbf{H}_B=S_z$. Hence the eigenstates of $\mathbf{H}_B$ must be eigenstates of $S_z$ and thus also of $S^2$ and, since the two operators are independent (they relate to two different types of spins, hence the operators should better commute) also to $I^2$ and $I_z$, if you want.
The crucial problem is that $S_z$ and $F^2$ do not commute. Why?
Well: $\mathbf{F}=\mathbf{I}+\mathbf{S}$ hence $F^2=S^2+I^2+2\mathbf{S}\cdot \mathbf{I}$. Now $S_z$ and $\mathbf{S}$ do not commute, because $S_z$ does not commute with e.g. $S_x$, which is part of $\mathbf{S}$. Since $F^2$ commutes with $\mathbf{H}_{hf}$ and $S_z$ commutes with $\mathbf{H}_B$, but not with $F^2$, we have that $\mathbf{H}_{hf}$ does not commute with $\mathbf{H}_B$. This means that $\mathbf{H}_B$ and $\mathbf{H}_{hf}$ cannot be diagonal in the same basis, hence you need to have off-diagonal elements.
In order to see how the matrix representing $\mathbf{H}_B$ looks like in the $|F^2,m_F\rangle$-basis, you can express the $|m_I,m_S\rangle$-basis (in which $\mathbf{H}_B$ is diagonal) in terms of the other basis. This is exactly what equations (4.21) do. These are obtained by ordinary addition of angular momenta. From there, you can construct the unitary transforming the basis $|m_I,m_S\rangle$ into $|F^2,m_F\rangle$ and $\mathbf{H}_B$ will be the diagonal matrix in the basis $|m_I,m_S\rangle$ conjugated with this unitary.
EDIT:
I'm not quite sure whether I understand correctly what your problem is, but let me elaborate: We want to find the Hamiltonian $\mathbf{H}_B$ in the $|m_Im_S\rangle$ basis. In this basis, it is diagonal, because $\mathbf{H}_B$ is essentially $S_z$ (hence commutes with $S_z$) and it must also commute with $I_z$ since $S_z$ and $I_z$ are independent.
If we order the basis according to $|\frac{1}{2},\frac{1}{2}\rangle,|-\frac{1}{2},-\frac{1}{2}\rangle,|\frac{1}{2},-\frac{1}{2}\rangle,|-\frac{1}{2},\frac{1}{2}\rangle$, then, we can just read off the Hamiltonian: The first and fourth vector are eigenvectors to eigenvalue $\mu B$, the others of $-\mu B$ (by definition of $S_z$, since the second component in $|m_Im_S\rangle=|(SI)I_z,S_z\rangle$ tells us the eigenvalue of $S_z$ that the basis vector corresponds to), i.e.
$$ \mathbf{H}_B=\begin{pmatrix}{} \mu B & 0 & 0 & 0 \\ 0 & -\mu B & 0 & 0 \\ 0 & 0 & -\mu B & 0 \\ 0 & 0 & 0 & \mu B\end{pmatrix}$$
Now, as I said, you just have to change the basis. The matrix transforming the above basis into the new basis is given by eqn. (4.21a-d):
$$U:=\begin{pmatrix}{} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & \frac{1}{2} & \frac{1}{2} \\ 0 & 0 & -\frac{1}{2} & \frac{1}{2} \end{pmatrix}$$
where the ordering of the $|Fm_F\rangle$-basis is as for $\mathbf{H}$ in your text.
Now calculate $U\mathbf{H}_B U^{\dagger}$ and that should give you the part of $\mathbf{H}$ coming from $\mathbf{H}_B$ in the $|F,m_F\rangle$-basis and this will be exactly what is written in your book.
EDIT 2: I sort of suspected this, so here is some more linear algebra for the problem. I'll use Dirac notion since I suspect you are more familiar with this:
Now suppose you have given two bases $|e_i\rangle$ and $|f_i\rangle$ and suppose they are orthonormal bases. What we want is a matrix $U$ that transforms one basis into the other (I'll call it $U$, since it'll be a unitary - if the bases are not orthonormal, it'll only be an invertible matrix). So we want a matrix such that
$$ |f_i\rangle:=U|e_i\rangle \qquad \forall i$$
How to construct this matrix? Well, given an equation for $|f_i\rangle$ in terms of the $|e_i\rangle$ will give you the i-th row of the matrix. You can also see the matrix elements in Dirac notation:
$$ \langle e_j|U|e_i\rangle=\langle e_j|f_i\rangle $$
In your case, $|e_i\rangle=|m_Im_S\rangle$ and $|f_i\rangle=|F^2,m_F\rangle$. Hence equation (4.21a) will give you the first row of the matrix (the ordering of the basis vectors $|m_Im_S\rangle$ as I proposed above), (4.21c) the second (notice the basis ordering in the matrix $\mathbf{H}$!) (4.21b) the second and (4.21d) the last row of the matrix. Using the equation for the matrix elements above, you should be able to check that with not too much trouble. You can also easily check that $U$ is indeed a unitary (i.e. $UU^{\dagger}=U^{\dagger}U=\mathbb{1}$.
Then we can calculate the matrix elements:
$$ \langle e_i |\mathbf{H}|e_j\rangle=\langle e_i|U^{\dagger}U\mathbf{H}U^{\dagger}U|e_j\rangle=\langle f_i| U\mathbf{H}U^{\dagger}|f_j\rangle $$, which tells you how the matrix looks like in the other basis.
Best Answer
You wouldn't think it, from how easy it is to pose this question, but it is ridiculously nontrivial. As it happens, it is entirely impossible to find the position-basis matrix elements of this propagator.
So far you've done good, and the identification $$ U\left( t_{2},t_{1}\right) =e^{\frac {-i} {\hbar }H\left( t_{2}-t_{1}\right) } =\sum_{n=-\infty}^\infty e^{-i\left( t_{2}-t_{1}\right) E_{n}/\hbar}\lvert n\rangle\langle n\rvert $$ is correct. Naively, the path is clear, because you know the eigenfunctions, $$ ⟨\theta|n⟩=\frac{1}{\sqrt{2\pi}}e^{in\theta}, $$ so you can in principle just sandwich $U(t_2,t_1)$ between two position kets and you'd be done. This gives you \begin{align} \langle \theta_2 | U\left( t_{2},t_{1}\right) | \theta_1\rangle & = \sum_{n=-\infty}^\infty e^{-i\left( t_{2}-t_{1}\right) E_{n}/\hbar}\langle \theta_2 | n\rangle\langle n|\theta_1\rangle \\ & = \frac{1}{2\pi} \sum_{n=-\infty}^\infty e^{-i\left( t_{2}-t_{1}\right) E_{n}/\hbar}e^{in(\theta_1-\theta_2)}, \end{align} and everything looks fine, right? You've just got one series to calculate and you'll be on your way. Moreover, you can simplify things here a fair bit: you can set $\theta=\theta_1-\theta_2$ and $\tau=t_2-t_1$ to simplify notation (i.e. the propagator only depends on those differences, which is nice), and you can also calculate the energies, $$ E_n=\frac{\hbar^2}{2mR^2}\left(n+\frac{e}{h}\phi\right)^2, $$ so that $ \frac{(t_2-t_1)E_n}{\hbar} = \frac{\hbar}{2mR^2}\left(n+\frac{e}{h}\phi\right)^2\tau =\kappa n^2\tau + \lambda n\phi\tau + \mu\phi^2\tau, $ and all of this makes for a nicely simplified series: \begin{align} \langle \theta_2 | U\left( t_{2},t_{1}\right) | \theta_1\rangle & = \frac{e^{-i\mu\phi^2\tau}}{2\pi} \sum_{n=-\infty}^\infty e^{-in^2\kappa \tau }e^{in(\theta-\lambda\phi\tau)}, \end{align} and this is clean and simple as can be. What could possibly go wrong?
Now, as it happens, this series is exactly summable, and it represents something called a Jacobi $\vartheta_3$ function, and you can read all about it in the DLMF . More precisely, in this case, the relationship comes down to \begin{align} \langle \theta_2 | U\left( t_{2},t_{1}\right) | \theta_1\rangle & = \frac{1}{\sqrt{4\pi i \kappa\tau}} e^{-i\mu\phi^2\tau} e^{\frac{i (\lambda\phi\tau-\theta)^2}{4 \kappa\tau}} \vartheta_3\mathopen{}\left( \frac{\pi }{2 \kappa\tau}(\lambda\phi\tau-\theta), e^{i\frac{ \pi^2}{\kappa\tau}} \right)\mathclose{} , \tag{$\ast$} \end{align} and it kinda looks like you're done: you've reduced your series to some special function, and you obviously cannot whittle that down any further, so there's nothing else to do.
Unfortunately, however, you're not actually done yet. If you look more closely at the theta function, it is normally defined as $$ \vartheta_3(z,q)=1+2\sum_{n=0}^\infty q^{n^2}\cos(2nz), $$ where $q$ and $z$ are complex parameters, and we will want to set $q=e^{i \pi^2/\kappa\tau}$. It is obvious from the definition that the series will converge if $|q|<1$, and that it will diverge if $|q|>1$. The quantum mechanics result we just got is balanced on the knife edge, at $|q|=1$: there are no convergence guarantees, but the series is oscillatory, so maybe we can hope for some conditional convergence brought about by the oscillations?
Alas, there's no joy there either. The full answer is that $\vartheta_3$ has something called a natural boundary, which means that for any point $q_0$ on the unit circle (so $|q_0|=1$) and any $z\in\mathbb C$, the limit $\lim_{q\to q_0^-} \vartheta_3(z,q)$ diverges to infinity. (I'm using the notation $q\to q_0^-$ to imply that $q$ stays inside the disk during the limit, just to keep things easy.) If you want the big ugly mathematician details, this is because the series is something called a lacunary series, which I asked about here on MathOverflow and here on Maths SE; in particular, here the Fabry gap theorem implies that the theta function has a ring of fire around it, which sort of looks like this:
Mathematica code through Import["https://raw.githubusercontent.com/halirutan/Mathematica-SE-Tools/master/SETools/SEImageExpressionDecode.m"]["http://i.stack.imgur.com/wM9dN.png"]
The problem here is that our $\vartheta_3$ is sitting right on top of the natural boundary: it is an analytical function for $|q|<1$, but there is provably no way to provide an analytical continuation past its circle of convergence (and that is kind of a bummer), and there is provably no way to make any sense of it for $|q|=1$, which is exactly where we are.
So... where does that leave us? Our nice propagator $(\ast)$ looked neat enough, but a closer inspection shows that it is actually mathematical nonsense. What does this mean, and how do we fix it?
Well, there's a couple of ways, but they both essentially mean that you are only allowed to interpret $\langle \theta_2 | U\left( t_{2},t_{1}\right) | \theta_2+\theta \rangle$ as something called a generalized function, also known as a distribution: an object, kind of like the Dirac delta function, that doesn't make sense on its own, but only when integrated in products with other functions.
There's two easy ways to do this:
One is to take the complex-$q$ thing seriously. This is easily achieved by giving the time $\tau=\tau_0-i\delta$ a nonzero imaginary component, which brings $|q|$ down to under $1$, and lets $⟨ \theta_2 | U\left( t_{2},t_{1}\right) | \theta_1 ⟩$ make sense.
If you do this, the key thing to note is that normally the only physically meaningful quantities where you will use the propagator are matrix elements of the form $⟨f|U(t_2, t_1)|g⟩$ for physical states $|f⟩$ and $|g⟩$. Those ultimately boil down to integrals of the form \begin{align} ⟨f|U(t_2, t_1)|g⟩ & = \int_0^{2\pi} ⟨0|U(t_2, t_1)|\theta⟩ F(\theta)\mathrm d\theta \\&= \int_0^{2\pi} \frac{ e^{-i\mu\phi^2\tau} e^{\frac{i (\lambda\phi\tau-\theta)^2}{4 \kappa\tau}} }{\sqrt{4\pi i \kappa\tau}} \vartheta_3\mathopen{}\left( \frac{\pi }{2 \kappa\tau}(\lambda\phi\tau-\theta), e^{i\frac{ \pi^2}{\kappa\tau}} \right)\mathclose{} F(\theta)\mathrm d\theta , \end{align} and here the recipe says that you should read those integrals as the limit of integrals with a nonzero $\delta$ as that $\delta$ goes to zero: \begin{align} ⟨f|U(t_2, t_1)|g⟩ & = \lim_{\delta\to 0} \int_0^{2\pi} \frac{ e^{-i\mu\phi^2\tau} e^{\frac{i (\lambda\phi\tau-\theta)^2}{4 \kappa\tau}} }{\sqrt{4\pi i \kappa\tau}} \vartheta_3\mathopen{}\left( \frac{\pi }{2 \kappa\tau}(\lambda\phi\tau-\theta), e^{i\pi^2/\kappa(\tau-i\delta)} \right)\mathclose{} F(\theta)\mathrm d\theta . \end{align} If you do that, what happens is that for each $\delta$ the integrand is well defined, analytical and perfectly well behaved, and it gives you a perfectly normal result for the integral. More importantly, those results will approach a limit as $\delta\to0$ as long as $F(\theta)$ is not absolutely crazy.
The other approach is similar, but it's series-based, i.e. it requires you to take our original series before the evaluation to $\vartheta_3$, \begin{align} \langle \theta_2 | U\left( t_{2},t_{1}\right) | \theta_1\rangle & = \frac{e^{-i\mu\phi^2\tau}}{2\pi} \sum_{n=-\infty}^\infty e^{-in^2\kappa \tau }e^{in(\theta-\lambda\phi\tau)}, \end{align} and see it as indicating that you need to take the $|n|\to\infty$ limit after you've done any relevant inner products you need to take. For the integral above, for example, this means the understanding that \begin{align} \int_0^{2\pi} ⟨0|U(t_2, t_1)|\theta⟩ F(\theta)\mathrm d\theta &= \frac{e^{-i\mu\phi^2\tau}}{2\pi} \sum_{n=-\infty}^\infty e^{-in^2\kappa \tau } \int_0^{2\pi} e^{in(\theta-\lambda\phi\tau)} F(\theta)\mathrm d\theta , \end{align} with the series after the integral. If $F(\theta)$ is well-behaved, then its Fourier series coefficients will go down with $|n|$ reasonably fast, and the series will (whew!) converge again - as it needs to do for a physically relevant matrix element.
Alternatively, you can take the series expression we had earlier, \begin{align} \langle \theta_2 | U\left( t_{1}+\tau,t_{1}\right) | \theta_2+\theta\rangle & = \frac{e^{-i\mu\phi^2\tau}}{2\pi} \sum_{n=-\infty}^\infty e^{-in^2\kappa \tau }e^{in(\theta-\lambda\phi\tau)}, \end{align} and simply see it as an identity between distributions.
This paradigm requires you to see each mapping $\theta\mapsto e^{-in^2\kappa \tau }e^{in(\theta-\lambda\phi\tau)}$ not as a function but as a distribution, which you are then adding up together as a series. This is a pretty strong thing to do, since it requires you to stop seeing $⟨\theta_2|n⟩⟨n|\theta_1⟩$ as something which has a value if you give it a specific $\theta_1$, $\theta_2$ and $n$, but if you do this you get some nice mathematical advantages.
More specifically, each of the distributions $\theta\stackrel{\mathrm{DS}}{\mapsto} e^{-in^2\kappa \tau }e^{in(\theta-\lambda\phi\tau)}$ is a tempered distribution (i.e. the kind of distribution that plays well with Fourier transforms and quantum mechanics; see these notes, these ones or these ones for more details), and the series is bounded above termwise by the Dirac comb, which is also a tempered distribution, and this is (probably?) plenty to show that the series sums to a tempered distribution.
Thus, in this paradigm, you get that the propagator exists and it is as nice as you'd like it to be (within the bounds that it is a distribution) and that it is indeed the limit of the series on the right hand side, so long as you take it in the distributional sense.
That's sort of it, really, and I think this is as clear as I can make it. I'll close with the two references where I learned this stuff,
and
and finally with a quote from Schulman (actually present in both references), which describes the situation here rather accurately: