Let us start from
$$
M_{ij}=\epsilon_{ijk}J_{k}\ ,\quad M_{0i}=-K_i
$$
(the minus sign is my convention, sorry!). If you know, a priori, that
\begin{align*}
[J_i,J_j]&=i\epsilon_{ijk}J_k\ ,\\
[K_i,K_j]&=-i\epsilon_{ijk}J_k\ ,\\
[J_i,K_j]&=i\epsilon_{ijk}K_j\ ,
\end{align*}
then you can just do a brute force computation for appropriate indices and figure out the pattern for the general case. For example
\begin{align*}
[M_{0i},M_{0j}]&=[K_i,K_j]=-i\epsilon_{ijk}J_k=-i\,M_{ij}\ ,
\end{align*}
and so on. Alternatively, you can write down $J$ explicitly in terms of $M$ by doing
$$
\epsilon_{ijq}M_{ij}=\epsilon_{ijq}\epsilon_{ijk}J_k=2\delta_{qk}J_k\Rightarrow J_k=\frac{1}{2}\epsilon_{kij}M_{ij}\ .
$$
Then we can substitute this directly on the commutation relation for $J$'s
$$
\frac{1}{4}[\epsilon_{mni}M_{mn},\epsilon_{pqj}M_{pq}]=i\epsilon_{ijk}\epsilon_{uvk}M_{uv}=i\left(\delta_{iu}\delta_{jv}-\delta_{iv}\delta_{ju}\right)M_{uv}=-2iM_{ij}\ .\quad (\Box)
$$
Then you must replace the product of Levi-Civita symbols on the LHS of $(\Box)$ by their representation of the product of $\delta$'s to get
$$
[M_{ik},M_{jk}]=-2iM_{ij}\ .
$$
You then carry on substituting $J$ on the commutation relation for $J$ with $K$, but in the end you still must guess the general pattern.
Unfortunately none of these approaches are very practical. If all you want is a way to find the commutation relations for the generators of the Lorentz group, I suggest writing them down on a particular representation, e.g.
$$
(M_{\mu\nu})^{\sigma}_{\ \ \rho}=i\left(\eta_{\mu\rho}\delta^\sigma_{\ \ \nu}-\eta_{\nu\rho}\delta^\sigma_{\ \ \mu}\right)\ .
$$
On the above expression, $\sigma$ and $\rho$ represent matrix indices sort of speak. Then one knows how $M_{\mu\nu}$ transforms under a Lorentz transformation
$$
\Lambda M_{\mu\nu}\Lambda^{-1}=M_{\lambda\sigma}\Lambda^{\lambda}_{\ \ \mu}\Lambda^{\sigma}_{\ \ \nu}.\quad (\star)
$$
Considering infinitesimal transformations $\Lambda=1-\frac{i}{2}\xi^{\lambda\sigma}M_{\lambda\sigma}$ (for some antisymmetric parameter $\xi$), the LHS of $(\star)$ can be written as
$$
\Lambda M_{\mu\nu}\Lambda^{-1}=M_{\mu\nu}+\frac{i}{2}\xi^{\lambda\sigma}[M_{\mu\nu},M_{\lambda\sigma}]+\mathcal{O}(\xi^2)\ ,
$$
while the RHS of $(\star)$ can be written as
$$
M_{\lambda\sigma}\Lambda^{\lambda}_{\ \mu}\Lambda^{\sigma}_{\ \nu}=M_{\mu\nu}-\frac{1}{2}\xi^{\lambda\sigma}\left(M_{\mu\lambda}\eta_{\nu\sigma}-M_{\mu\sigma}\eta_{\nu\lambda}+M_{\lambda\nu}\eta_{\mu\sigma}-M_{\sigma\nu}\eta_{\mu\lambda}\right)+\mathcal{O}(\xi^2)\ ,
$$
thus implying
$$
[M_{\mu\nu},M_{\lambda\sigma}]=i\left(M_{\mu\lambda}\eta_{\nu\sigma}-M_{\mu\sigma}\eta_{\nu\lambda}+M_{\lambda\nu}\eta_{\mu\sigma}-M_{\sigma\nu}\eta_{\mu\lambda}\right)\ .
$$
Hope this helps! Cheers.
It's pretty annoying that P&S just give you
$$S^{\mu \nu} = \frac{i}{4} [\gamma^{\mu},\gamma^{\nu}]$$
from thin air, here is a way to derive it similar to Bjorken-Drell's derivation (who start from the Dirac equation) but from the Clifford algebra directly, assuming that products of the gamma matrices form a basis. Given a Clifford algebra of $\gamma^{\mu}$'s satisfying
\begin{align}
\{ \gamma^{\mu} , \gamma^{\mu} \} = 2 \eta^{\mu \nu} I
\end{align}
we note that for an invertible transformation $S$ we have
\begin{align}
2 \eta^{\mu \nu} I &= 2 \eta^{\mu \nu} S^{-1} S \\
&= S^{-1}(2 \eta^{\mu \nu}) S \\
&= S^{-1}\{ \gamma^{\mu} , \gamma^{\mu} \} S \\
&= \{ S^{-1} \gamma^{\mu} S, S^{-1}\gamma^{\mu} S \} \\
&= \{ \gamma'^{\mu} , \gamma'^{\mu} \}
\end{align}
showing us that the Clifford algebra of matrices
$$\gamma'^{\mu} = S^{-1} \gamma^{\mu} S$$
also satisfies the Clifford algebra, hence any set of matrices satisfying the Clifford algebra can be obtained from a given set $\gamma^{\mu}$ using a non-singular transformation $S$. Since the anti-commutation relations involve the metric $\eta_{\mu \nu}$, and we know the metric is left invariant under Lorentz transformations
$$\eta^{\mu \nu} = \Lambda^{\mu} \, _{\rho} \Lambda^{\nu} \, _{\sigma} \eta^{\rho \sigma} $$
this immediately implies
\begin{align}
2 \eta^{\mu \nu} I &= \Lambda^{\mu} \, _{\rho} \Lambda^{\nu} \, _{\sigma} 2 \eta^{\rho \sigma} I \\
&= \Lambda^{\mu} \, _{\rho} \Lambda^{\nu} \, _{\sigma} \{ \gamma^{\rho} , \gamma^{\sigma} \} \\
&= \{ \Lambda^{\mu} \, _{\rho} \gamma^{\rho} , \Lambda^{\nu} \, _{\sigma} \gamma^{\sigma} \} \\
&= \{ \gamma'^{\mu} , \gamma'^{\mu} \}
\end{align}
which shows that the Lorentz transformation of a gamma matrix also satisfies the Clifford algebra, and so is itself a gamma matrix, and hence can be expressed in terms of some non-singular transformation $S$
\begin{align}
\gamma'^{\mu} &= \Lambda^{\mu} \, _{\nu} \gamma^{\nu} \\
&= S^{-1} \gamma^{\mu} S
\end{align}
where $S$ is to be determined. Since the operators $S$ represent performing a Lorentz transformation on $\gamma^{\mu}$, and Lorentz transformations on fields expand as $I - \frac{i}{2}\omega_{\mu \nu} M^{\mu \nu}$, we expand $\Lambda$ and $S$ as
\begin{align}
\Lambda^{\mu} \, _{\nu} &= \delta ^{\mu} \, _{\nu} + \omega^{\mu} \, _{\nu} \\
S &= I - \frac{i}{2} \omega_{\mu \nu} \Sigma^{\mu \nu}
\end{align}
where $\Sigma^{\mu \nu}$ must be anti-symmetric and constructed from a basis of gamma matrices, hence from
\begin{align}
\gamma'^a &= \Lambda^a \, _{\mu} \gamma^{\mu} \\
&= (\delta^a \, _{\mu} + \omega^a \, _{\mu})\gamma^{\mu} \\
&= \gamma^a + \omega^a \, _{\mu} \gamma^{\mu} \\
&= \gamma^a + \omega_{b \mu} \eta^{a b} \gamma^{\mu} \\
&= \gamma^a + \omega_{b \mu} \eta^{a [b} \gamma^{\mu]} \\
&= \gamma^a + \frac{1}{2} \omega_{b \mu} (\eta^{a b} \gamma^{\mu} - \eta^{a \mu} \gamma^b) \\
&= \gamma^a + \frac{1}{2} \omega_{\nu} (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}) \\
&= S^{-1} \gamma^a S \\
&= (I - \frac{i}{2} \omega_{\mu \nu} \Sigma^{\mu \nu}) \gamma^a (I + \frac{i}{2} \omega_{\mu \nu} \Sigma^{\mu \nu}) \\
&= \gamma^a - \frac{i}{2} \omega_{\mu \nu} [\gamma^a, \Sigma^{\mu \nu}]
\end{align}
we have the relation (which can be interpreted as saying that $\gamma^a$ transforms as a vector under spinor representations of Lorentz transformations, as e.g. in Tong's QFT notes)
\begin{align}
i (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}) = [\gamma^a, \Sigma^{\mu \nu}]
\end{align}
and we know $\Sigma^{\mu \nu}$, since it is anti-symmetric, must involve a product's of $\gamma$ matrices (because of the 16-dimensional basis formed from Clifford algebra elements), only two by the left-hand side, and from
\begin{align}
\gamma^{\mu} \gamma^{\nu} &= - \gamma^{\nu} \gamma^{\mu} , \ \ \ \mu \neq \nu, \\
\gamma^{\mu} \gamma^{\mu} &= \gamma^{\nu} \gamma^{\mu} , \ \ \ \mu = \nu,
\end{align}
we expect that
\begin{align}
\Sigma^{\mu \nu} &= c [\gamma^{\mu},\gamma^{\nu}] \\
&= c (\gamma^{\mu} \gamma^{\nu} - \gamma^{\nu} \gamma^{\mu}) \\
&= 2 c ( \gamma^{\mu} \gamma^{\nu} - \eta^{\mu \nu})
\end{align}
for some $c$ which we constrain by the (vector) relation above
\begin{align}
i (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}) &= [\gamma^a, \Sigma^{\mu \nu}] \\
&= c [\gamma^a, 2( \gamma^{\mu} \gamma^{\nu} - \eta^{\mu \nu})] \\
&= 2 c [\gamma^a, \gamma^{\mu} \gamma^{\nu}] \\
&= 2 c ( \gamma^{\mu} [\gamma^a,\gamma^{\nu}] + [\gamma^a, \gamma^{\mu}] \gamma^{\nu}) \\
&= 2 c [ \gamma^{\mu} 2( \gamma^a \gamma^{\nu} - \eta^{a \nu}) + 2( \gamma^a \gamma^{\mu} - \eta^{a \mu}) \gamma^{\nu}] \\
&= 4 c [ \gamma^{\mu} ( \gamma^a \gamma^{\nu} - \eta^{a \nu}) + ( \gamma^{\mu} \gamma^{a} + 2 \eta^{a \mu} - \eta^{a \mu}) \gamma^{\nu}] \\
&= 4 c (\eta^{a \mu} \gamma^{\nu} - \eta^{a \nu} \gamma^{\mu}).
\end{align}
This gives the result $c = i/4$. The generator of Lorentz transformations of gamma matrices is
\begin{align}
\Sigma^{\mu \nu} &= \dfrac{i}{4} [\gamma^{\mu},\gamma^{\nu}] \\
&= \dfrac{i}{2}(\gamma^{\mu} \gamma^{\nu} - \eta^{\mu \nu}) \ \ \text{i.e.} \\
S &= I - \frac{i}{2} \omega_{\mu \nu} (\frac{i}{4} [\gamma^{\mu},\gamma^{\nu}]) \\
&= I + \dfrac{1}{8} \omega_{\mu \nu} [\gamma^{\mu},\gamma^{\nu}].
\end{align}
Using the fact that the gamma matrices transform as a vector under the spinor representation of an infinitesimal Lorentz transformation,
\begin{align}
[\Sigma^{\mu \nu}, \gamma^{\rho}] = i (\gamma^{\mu} \eta^{\nu \rho} - \gamma^{\nu} \eta^{\mu \rho})
\end{align}
we can show the spinor representation of a Lorentz transformation satisfies the Lorentz algebra commutation relations, since for $\rho \neq \sigma$
\begin{align}
[\Sigma^{\mu \nu},\Sigma^{\rho \sigma}] &= \frac{i}{2}[\Sigma^{\mu \nu},\gamma^{\rho} \gamma^{\sigma}] \\
&= \frac{i}{2}([\Sigma^{\mu \nu},\gamma^{\rho} ] \gamma^{\sigma} + \gamma^{\rho} [\Sigma^{\mu \nu}, \gamma^{\sigma}]) \\
&= \frac{i}{2}\{ i (\gamma^{\mu} \eta^{\nu \rho} - \gamma^{\nu} \eta^{\mu \rho}) \gamma^{\sigma} + \gamma^{\rho} i (\gamma^{\mu} \eta^{\nu \sigma} - \gamma^{\nu} \eta^{\mu \sigma}) \} \\
&= - \frac{1}{2}\{ \gamma^{\mu} \eta^{\nu \rho} \gamma^{\sigma} - \gamma^{\nu} \eta^{\mu \rho} \gamma^{\sigma} + \gamma^{\rho} \gamma^{\mu} \eta^{\nu \sigma} - \gamma^{\rho} \gamma^{\nu} \eta^{\mu \sigma} \} \\
&= \frac{i}{2}\{ \eta^{\nu \rho} (2 \Sigma^{\mu \sigma} + \eta^{\mu \sigma}) - \eta^{\mu \rho} (2 \Sigma^{\nu \sigma} - \eta^{\nu \sigma}) + (2 \Sigma^{\rho \mu} - \eta^{\rho \mu}) \eta^{\nu \sigma} - (2 \Sigma^{\rho \nu}) - \eta^{\rho \nu}) \eta^{\mu \sigma} \} \\
&= i ( \eta^{\nu \rho} \Sigma^{\mu \sigma} - \eta^{\mu \rho} \Sigma^{\nu \sigma} + \Sigma^{\rho \mu} \eta^{\nu \sigma} - \Sigma^{\rho \nu} \eta^{\mu \sigma} ).
\end{align}
This method generalizes from $SO(3,1)$ to $SO(N)$, see e.g. Kaku QFT Sec. 2.6, and the underlying reason for doing any of this in the first place is that one seeks to find projective representations which arise due to the non-simple-connectedness of these orthogonal groups. Regarding your question about arbitrary metrics $g_{\mu \nu}$, this method applies to, and arises due to the non-simple-connectedness of, special orthogonal groups, you can't generalize to arbitrary metrics, this is a problem which can be circumvented in supergravity and superstring theory using veilbein's.
References:
- Bjorken, J.D. and Drell, S.D., 1964. Relativistic quantum mechanics; Ch. 2.
- Kaku, M., 1993. Quantum field theory: a modern introduction. Oxford Univ. Press; Sec. 2.6.
- Tong, Quantum Field Theory Notes http://www.damtp.cam.ac.uk/user/tong/qft.html.
- http://www.damtp.cam.ac.uk/user/examples/D18S.pdf
- Does $GL(N,\mathbb{R})$ own spinor representation? Which group is its covering group? (Kaku's QFT textbook)
Best Answer
The metric appears because \begin{equation} \partial^\mu x^\nu = \partial^\mu g^{\rho\nu} x_\rho = g^{\rho\nu} \partial^\mu x_\rho = g^{\rho\nu} \delta^\mu_\rho = g^{\mu\nu} \end{equation} (write out the meaning of $\partial^\mu$ to see why this must be so). Using this relation, it is straightforward to derive equation (3.17) from (3.16).
As to whether the $J^{\mu\nu}$ are Lorentz transformations, ZeroTheHero is entirely correct; using that \begin{equation} \exp(-i\omega_{\mu\nu}J^{\mu\nu}) := \sum_{n=0}^\infty \frac{(-i\omega_{\mu\nu}J^{\mu\nu})^n}{n!} = \lim_{n\to\infty} \left(1 - \frac{i\omega_{\mu\nu}J^{\mu\nu}}{n}\right)^n , \tag{1} \end{equation} (with some restrictions on $\omega_{\mu\nu}, J^{\mu\nu}$) it can be shown that the generators $J^{\mu\nu}$ "generates" a representation of the Lorentz group through repeated multiplication/composition [cf. the last equality in (1)].