UPDATE - Answer edited to be consistent with the latest version of the question.
The different definitions you mentioned are NOT definitions. In fact, what you are describing are different representations of the Lorentz Algebra. Representation theory plays a very important role in physics.
As far as the Lie algebra are concerned, the generators $L_{\mu\nu}$ are simply some operators with some defined commutation properties.
The choices $L_{\mu\nu} = J_{\mu\nu}, S_{\mu\nu}$ and $M_{\mu\nu}$ are different realizations or representations of the same algebra. Here, I am defining
\begin{align}
\left( J_{\mu\nu} \right)_{ab} &= - i \left( \eta_{\mu a} \eta_{\nu b} - \eta_{\mu b} \eta_{\nu a} \right) \\
\left( S_{\mu\nu}\right)_{ab} &= \frac{i}{4} [ \gamma_\mu , \gamma_\nu ]_{ab} \\
M_{\mu\nu} &= i \left( x_\mu \partial_\nu + x_\nu \partial_\mu \right)
\end{align}
Another possible representation is the trivial one, where $L_{\mu\nu}=0$.
Why is it important to have these different representations?
In physics, one has several different fields (denoting particles). We know that these fields must transform in some way under the Lorentz group (among other things). The question then is, How do fields transform under the Lorentz group? The answer is simple. We pick different representations of the Lorentz algebra, and then define the fields to transform under that representation! For example
- Objects transforming under the trivial representation are called scalars.
- Objects transforming under $S_{\mu\nu}$ are called spinors.
- Objects transforming under $J_{\mu\nu}$ are called vectors.
One can come up with other representations as well, but these ones are the most common.
What about $M_{\mu\nu}$ you ask? The objects I described above are actually how NON-fields transform (for lack of a better term. I am simply referring to objects with no space-time dependence). On the other hand, in physics, we care about FIELDS. In order to describe these guys, one needs to define not only the transformation of their components but also the space time dependences. This is done by including the $M_{\mu\nu}$ representation to all the definitions described above. We then have
- Fields transforming under the trivial representation $L_{\mu\nu}= 0 + M_{\mu\nu}$ are called scalar fields.
- Fields transforming under $S_{\mu\nu} + M_{\mu\nu} $ are called spinor fields.
- Fields transforming under $J_{\mu\nu} + M_{\mu\nu}$ are called vector fields.
Mathematically, nothing makes these representations any more fundamental than the others. However, most of the particles in nature can be grouped into scalars (Higgs, pion), spinors (quarks, leptons) and vectors (photon, W-boson, Z-boson). Thus, the above representations are often all that one talks about.
As far as I know, Clifford Algebras are used only in constructing spinor representations of the Lorentz algebra. There maybe some obscure context in some other part of physics where this pops up, but I haven't seen it. Of course, I am no expert in all of physics, so don't take my word for it. Others might have a different perspective of this.
Finally, just to be explicit about how fields transform (as requested) I mention it here. A general field $\Phi_a(x)$ transforms under a Lorentz transformation as
$$
\Phi_a(x) \to \sum_b \left[ \exp \left( \frac{i}{2} \omega^{\mu\nu} L_{\mu\nu} \right) \right]_{ab} \Phi_b(x)
$$
where $L_{\mu\nu}$ is the representation corresponding to the type of field $\Phi_a(x)$ and $\omega^{\mu\nu}$ is the parameter of the Lorentz transformation. For example, if $\Phi_a(x)$ is a spinor, then
$$
\Phi_a(x) \to \sum_b \left[ \exp \left( \frac{i}{2} \omega^{\mu\nu} \left( S_{\mu\nu} + M_{\mu\nu} \right) \right) \right]_{ab} \Phi_b(x)
$$
It's pretty annoying that P&S just give you
$$S^{\mu \nu} = \frac{i}{4} [\gamma^{\mu},\gamma^{\nu}]$$
from thin air, here is a way to derive it similar to Bjorken-Drell's derivation (who start from the Dirac equation) but from the Clifford algebra directly, assuming that products of the gamma matrices form a basis. Given a Clifford algebra of $\gamma^{\mu}$'s satisfying
\begin{align}
\{ \gamma^{\mu} , \gamma^{\mu} \} = 2 \eta^{\mu \nu} I
\end{align}
we note that for an invertible transformation $S$ we have
\begin{align}
2 \eta^{\mu \nu} I &= 2 \eta^{\mu \nu} S^{-1} S \\
&= S^{-1}(2 \eta^{\mu \nu}) S \\
&= S^{-1}\{ \gamma^{\mu} , \gamma^{\mu} \} S \\
&= \{ S^{-1} \gamma^{\mu} S, S^{-1}\gamma^{\mu} S \} \\
&= \{ \gamma'^{\mu} , \gamma'^{\mu} \}
\end{align}
showing us that the Clifford algebra of matrices
$$\gamma'^{\mu} = S^{-1} \gamma^{\mu} S$$
also satisfies the Clifford algebra, hence any set of matrices satisfying the Clifford algebra can be obtained from a given set $\gamma^{\mu}$ using a non-singular transformation $S$. Since the anti-commutation relations involve the metric $\eta_{\mu \nu}$, and we know the metric is left invariant under Lorentz transformations
$$\eta^{\mu \nu} = \Lambda^{\mu} \, _{\rho} \Lambda^{\nu} \, _{\sigma} \eta^{\rho \sigma} $$
this immediately implies
\begin{align}
2 \eta^{\mu \nu} I &= \Lambda^{\mu} \, _{\rho} \Lambda^{\nu} \, _{\sigma} 2 \eta^{\rho \sigma} I \\
&= \Lambda^{\mu} \, _{\rho} \Lambda^{\nu} \, _{\sigma} \{ \gamma^{\rho} , \gamma^{\sigma} \} \\
&= \{ \Lambda^{\mu} \, _{\rho} \gamma^{\rho} , \Lambda^{\nu} \, _{\sigma} \gamma^{\sigma} \} \\
&= \{ \gamma'^{\mu} , \gamma'^{\mu} \}
\end{align}
which shows that the Lorentz transformation of a gamma matrix also satisfies the Clifford algebra, and so is itself a gamma matrix, and hence can be expressed in terms of some non-singular transformation $S$
\begin{align}
\gamma'^{\mu} &= \Lambda^{\mu} \, _{\nu} \gamma^{\nu} \\
&= S^{-1} \gamma^{\mu} S
\end{align}
where $S$ is to be determined. Since the operators $S$ represent performing a Lorentz transformation on $\gamma^{\mu}$, and Lorentz transformations on fields expand as $I - \frac{i}{2}\omega_{\mu \nu} M^{\mu \nu}$, we expand $\Lambda$ and $S$ as
\begin{align}
\Lambda^{\mu} \, _{\nu} &= \delta ^{\mu} \, _{\nu} + \omega^{\mu} \, _{\nu} \\
S &= I - \frac{i}{2} \omega_{\mu \nu} \Sigma^{\mu \nu}
\end{align}
where $\Sigma^{\mu \nu}$ must be anti-symmetric and constructed from a basis of gamma matrices, hence from
\begin{align}
\gamma'^a &= \Lambda^a \, _{\mu} \gamma^{\mu} \\
&= (\delta^a \, _{\mu} + \omega^a \, _{\mu})\gamma^{\mu} \\
&= \gamma^a + \omega^a \, _{\mu} \gamma^{\mu} \\
&= \gamma^a + \omega_{b \mu} \eta^{a b} \gamma^{\mu} \\
&= \gamma^a + \omega_{b \mu} \eta^{a [b} \gamma^{\mu]} \\
&= \gamma^a + \frac{1}{2} \omega_{b \mu} (\eta^{a b} \gamma^{\mu} - \eta^{a \mu} \gamma^b) \\
&= \gamma^a + \frac{1}{2} \omega_{\nu} (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}) \\
&= S^{-1} \gamma^a S \\
&= (I - \frac{i}{2} \omega_{\mu \nu} \Sigma^{\mu \nu}) \gamma^a (I + \frac{i}{2} \omega_{\mu \nu} \Sigma^{\mu \nu}) \\
&= \gamma^a - \frac{i}{2} \omega_{\mu \nu} [\gamma^a, \Sigma^{\mu \nu}]
\end{align}
we have the relation (which can be interpreted as saying that $\gamma^a$ transforms as a vector under spinor representations of Lorentz transformations, as e.g. in Tong's QFT notes)
\begin{align}
i (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}) = [\gamma^a, \Sigma^{\mu \nu}]
\end{align}
and we know $\Sigma^{\mu \nu}$, since it is anti-symmetric, must involve a product's of $\gamma$ matrices (because of the 16-dimensional basis formed from Clifford algebra elements), only two by the left-hand side, and from
\begin{align}
\gamma^{\mu} \gamma^{\nu} &= - \gamma^{\nu} \gamma^{\mu} , \ \ \ \mu \neq \nu, \\
\gamma^{\mu} \gamma^{\mu} &= \gamma^{\nu} \gamma^{\mu} , \ \ \ \mu = \nu,
\end{align}
we expect that
\begin{align}
\Sigma^{\mu \nu} &= c [\gamma^{\mu},\gamma^{\nu}] \\
&= c (\gamma^{\mu} \gamma^{\nu} - \gamma^{\nu} \gamma^{\mu}) \\
&= 2 c ( \gamma^{\mu} \gamma^{\nu} - \eta^{\mu \nu})
\end{align}
for some $c$ which we constrain by the (vector) relation above
\begin{align}
i (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}) &= [\gamma^a, \Sigma^{\mu \nu}] \\
&= c [\gamma^a, 2( \gamma^{\mu} \gamma^{\nu} - \eta^{\mu \nu})] \\
&= 2 c [\gamma^a, \gamma^{\mu} \gamma^{\nu}] \\
&= 2 c ( \gamma^{\mu} [\gamma^a,\gamma^{\nu}] + [\gamma^a, \gamma^{\mu}] \gamma^{\nu}) \\
&= 2 c [ \gamma^{\mu} 2( \gamma^a \gamma^{\nu} - \eta^{a \nu}) + 2( \gamma^a \gamma^{\mu} - \eta^{a \mu}) \gamma^{\nu}] \\
&= 4 c [ \gamma^{\mu} ( \gamma^a \gamma^{\nu} - \eta^{a \nu}) + ( \gamma^{\mu} \gamma^{a} + 2 \eta^{a \mu} - \eta^{a \mu}) \gamma^{\nu}] \\
&= 4 c (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}).
\end{align}
This gives the result $c = i/4$. The generator of Lorentz transformations of gamma matrices is
\begin{align}
\Sigma^{\mu \nu} &= \dfrac{i}{4} [\gamma^{\mu},\gamma^{\nu}] \\
&= \dfrac{i}{2}(\gamma^{\mu} \gamma^{\nu} - \eta^{\mu \nu}) \ \ \text{i.e.} \\
S &= I - \frac{i}{2} \omega_{\mu \nu} (\frac{i}{4} [\gamma^{\mu},\gamma^{\nu}]) \\
&= I + \dfrac{1}{8} \omega_{\mu \nu} [\gamma^{\mu},\gamma^{\nu}].
\end{align}
Using the fact that the gamma matrices transform as a vector under the spinor representation of an infinitesimal Lorentz transformation,
\begin{align}
[\Sigma^{\mu \nu}, \gamma^{\rho}] = i (\gamma^{\mu} \eta^{\nu \rho} - \gamma^{\nu} \eta^{\mu \rho})
\end{align}
we can show the spinor representation of a Lorentz transformation satisfies the Lorentz algebra commutation relations, since for $\rho \neq \sigma$
\begin{align}
[\Sigma^{\mu \nu},\Sigma^{\rho \sigma}] &= \frac{i}{2}[\Sigma^{\mu \nu},\gamma^{\rho} \gamma^{\sigma}] \\
&= \frac{i}{2}([\Sigma^{\mu \nu},\gamma^{\rho} ] \gamma^{\sigma} + \gamma^{\rho} [\Sigma^{\mu \nu}, \gamma^{\sigma}]) \\
&= \frac{i}{2}\{ i (\gamma^{\mu} \eta^{\nu \rho} - \gamma^{\nu} \eta^{\mu \rho}) \gamma^{\sigma} + \gamma^{\rho} i (\gamma^{\mu} \eta^{\nu \sigma} - \gamma^{\nu} \eta^{\mu \sigma}) \} \\
&= - \frac{1}{2}\{ \gamma^{\mu} \eta^{\nu \rho} \gamma^{\sigma} - \gamma^{\nu} \eta^{\mu \rho} \gamma^{\sigma} + \gamma^{\rho} \gamma^{\mu} \eta^{\nu \sigma} - \gamma^{\rho} \gamma^{\nu} \eta^{\mu \sigma} \} \\
&= \frac{i}{2}\{ \eta^{\nu \rho} (2 \Sigma^{\mu \sigma} + \eta^{\mu \sigma}) - \eta^{\mu \rho} (2 \Sigma^{\nu \sigma} - \eta^{\nu \sigma}) + (2 \Sigma^{\rho \mu} - \eta^{\rho \mu}) \eta^{\nu \sigma} - (2 \Sigma^{\rho \nu}) - \eta^{\rho \nu}) \eta^{\mu \sigma} \} \\
&= i ( \eta^{\nu \rho} \Sigma^{\mu \sigma} - \eta^{\mu \rho} \Sigma^{\nu \sigma} + \Sigma^{\rho \mu} \eta^{\nu \sigma} - \Sigma^{\rho \nu} \eta^{\mu \sigma} ).
\end{align}
This method generalizes from $SO(3,1)$ to $SO(N)$, see e.g. Kaku QFT Sec. 2.6, and the underlying reason for doing any of this in the first place is that one seeks to find projective representations which arise due to the non-simple-connectedness of these orthogonal groups. Regarding your question about arbitrary metrics $g_{\mu \nu}$, this method applies to, and arises due to the non-simple-connectedness of, special orthogonal groups, you can't generalize to arbitrary metrics, this is a problem which can be circumvented in supergravity and superstring theory using veilbein's.
References:
- Bjorken, J.D. and Drell, S.D., 1964. Relativistic quantum mechanics; Ch. 2.
- Kaku, M., 1993. Quantum field theory: a modern introduction. Oxford Univ. Press; Sec. 2.6.
- Tong, Quantum Field Theory Notes http://www.damtp.cam.ac.uk/user/tong/qft.html.
- http://www.damtp.cam.ac.uk/user/examples/D18S.pdf
- Does $GL(N,\mathbb{R})$ own spinor representation? Which group is its covering group? (Kaku's QFT textbook)
Best Answer
Well, let us consider the fundamental representation of $SO(1,3)_+$ made of matrices $$\Lambda = [{\Lambda^a}_b]_{a,b=0,1,2,3}\:.$$ The position of indices is of crucial relevance here. The Lorentz-group condition is $$\Lambda^t \eta \Lambda = \eta$$ namely $${\Lambda^a}_b \eta_{ac} {\Lambda^c}_d = \eta_{bd}\:.$$ Let us expand around the identity the matrices and let us drop second order terms: $$(\delta^a_b + {\omega^a}_b)\eta_{ac}(\delta^c_d+ {\omega^c}_d) = \eta_{bd}$$ obtaining $$\delta^a_b \eta_{ac} {\omega^c}_d+ {\omega^a}_b\eta_{ac} \delta^c_d=0\:.$$ In other words $$\eta_{bc} {\omega^c}_d+ {\omega^a}_b\eta_{ad} =0\:.\tag{1}$$ There are $6$ linearly independent real $4\times 4$ matrices $\omega = [{\omega^a}_b]_{a,b=0,1,2,3}$ satisfying these identities, they form a basis of the Lie algebra of $SO(1,3)_+$.
In particular, the three standard boost generators written as $4\times 4$ matrices $$K_k= [{(K_k)^a}_b]_{a,b=0,1,2,3}, \quad k=x,y,z$$ do satisfy this identity!
The same fact is true for the three remaining generators of spatial rotations $S_k$, $k=x,y,z$.
The $6$ matrices $K_k$ and $S_k$ are also linearly independent so that they form a basis of the considered Lie algebra.
However, defining $$\omega_{ab}:= \eta_{ac} {\omega^c}_b$$ (1) can be equivalently restated as $$\omega_{ab}+ \omega_{ba}=0\:.\tag{2}$$ Notice that, as $\eta\eta = I$, the components ${\omega^a}_b$, and $\omega_{ab}$ carry the same information and one passes from a type to another simply exploiting the standard procedure of rising and lowering indices (though this procedure should be justified properly viewing the Lorentz transformations as $(1,1)$ tensors).
From an elementary viewpoint (1) are the definition of the generators of the Lie algebra of $SO(1,3)_+$, (2) are equivalent statements which should be handled cum grano salis.
The same procedure for $SU(2)$, writing the matrices as $U = [{U^a}_b]_{a,b=1,2}$, the unitarity condition produces $$\overline{{\omega^a}_b}+ {\omega^b}_a =0\:,$$ in place of (1), where the bar denotes the complex conjugation. Here a basis of the Lie algebra is made of the standard Pauli matrices with immaginary coefficient $-i\sigma_k = -i[{(\sigma_k)^a}_b]_{a,b =1,2}$, where $k=1,2,3$.Here we have also to impose that the element of the Lie algebra are traceless to fulfill the requirement $\det U=1$.