I don't understand why it is said that the (rest) mass of a system is not conserved in relativity.
I mean, the momentum of a system is conserved (i.e.: it remains constant in a frame of reference without any external influence).
Also the energy of a system is conserved in relativity: it doesn't change without any external agency
and the (rest) mass of a system is just
$$m^2=E^2-p^2$$
$E$ is constant, $p$ is constant, how can $m$ change?
For example, entropy is not conserved in a system, that means that the entropy of the system will increase spontaneously with time, which is really the case. But, is this true for mass?
here is the problem from Griffiths and exact solution as given in the Griffiths,
Two lumps of clay, each of rest mass $m$, collide head-on at $3c/5$ and they stick together. Question: what is the rest mass $M$ of the composite lump?
Solution: In this case conservation of momentum is trivial it is zero before and zero after. The energy of each lump prior to the collision is
$$mc^2/\sqrt{1-v^2/c^2} =5mc^2/4.$$ The energy of the composite lump after the collision is $Mc^2$ (since its at rest).
so the conservation of energy says:
$5mc^2/4 + 5mc^2/4 =Mc^2$ and hence
$M=5m/2$.
Notice that this is greater than the sum of the initial masses! Mass was not conserved in this collision; kinetic energy was converted into rest energy, so the rest mass increased.
Best Answer
Mass, or more correctly, rest mass is not conserved in special relativity. Particles are able to be created and annihilated in special relativity, for instance, an electron and a positron can interact to produce two photons: $$e^++e^-\rightarrow 2\gamma $$ Here mass is clearly not conserved, because both the electron and positron are massive but photons are massless.
In more detail, take the situation where $e^+$ and $e^-$ have opposite momentum, $p_0$. The total energy of the system is then: $$E_T^2 = 2m_e+2p_0^2$$ where $m_e$ is the electron/positron mass. To conserve energy-momentum in the collision, we require that the photons afterwards have equal and opposite momenta $p_1$, and also that $$E_T^2 = 2p_1^2$$ which means that $$p_1 = \sqrt{m_e+p_0^2}.$$ On the other hand, the electron and positron both have mass $m_e$, but the two photons are massless, and hence mass is not conserved.
The error in your reasoning in the question is that the formula $$m^2 = E^2 - p^2$$ only holds for the energy, mass, and momentum of a single particle, and so does not work when you are talking about the energy and momentum of a system as a whole.