The center of mass 4-momentum is the sum of the 4-momenta of the particles (no vector symbol or index, but the v's are four-component vectors) using the masses as the weights:
$$ P_\mathrm{CM} = m_1 v_1 + m_2 v_2 $$
The length of this is the mass of the combined system, (mostly minus metric)
$$ M^2 = |P|^2 = m_1^2 + m_2^2 + 2m_1 m_2 v_1 \cdot v_2 $$
The four-velocity of the center of mass is then
$$ v_\mathrm{CM} = {m_1v_1 + m_2 v_2 \over M} $$
and the three velocity is given by the ratio of the space-components of the four vector to the time component:
$$ v^0_\mathrm{CM} = {m_1\gamma_1 + m_2 \gamma_2 \over M}$$
So that the center of mass velocity is:
$$ \vec{v}_\mathrm{CM} = {m_1\gamma_1 \vec{v}_1 + m_2\gamma_2 \vec{v}_2 \over m_1\gamma_1 + m_2\gamma_2}$$
or the weighted average of the velocities using the relativistic mass (the energy). This formula usually appears with energy letters replacing mass letters:
$$ \vec{v}_\mathrm{CM} = { E_1 \vec{v}_1 + E_2 \vec{v}_2 \over E_1 + E_2}$$
Where m_1 and m_2 are the masses, $v_1$ and $v_2$ are the 4-velocities, $E_1$ and $E_2$ are the energies, $\gamma_1 = {1\over \sqrt{1-|\vec v_1|^2}}$ and similarly for $\gamma_2$.
I have a particle of rest mass $M$, total energy $E$ colliding with a stationery particle of rest mass $m$
The combined total energy $\mathscr E$ of this entire two-particle system (wrt. the "reference lab system" in which the particle of mass $m$ is stationary) is accordingly $$\mathscr E = E + m~c^2,$$
and the combined total momentum
$$\mathscr P = \sqrt{ (E/c)^2 - (M~c)^2 } + 0.$$
I have to show that [...] the total energy, $E'$ in the frame where their centre of mass is at rest is given by: [...]
For this I find helpful to think of the two-particle system described above as being replaced by just one (ficticious) particle of (suitable) mass $\mathscr M$, which carries the same total energy $\mathscr E$ and the same total momentum $\mathscr P$.
Obviously therefore $\mathscr M^2~c^4 = \mathscr E^2 - \mathscr P^2~c^2$.
Now, the point to note is that this (invariant) mass $\mathscr M$ is exactly the equivalent of the " total energy, $E'$ in the frame where their centre of mass is at rest" that you're seeking;
$$E' := \mathscr M~c^2.$$
Or in other words: the combined four-momentum vector of the two-particle system, as expressed in its center-of-mass frame (i.e. where its three-momentum vector is exactly zero) is equivalently
$ \{ \mathscr M~c, 0, 0, 0 \} = \{ E'/c, 0, 0, 0 \} $.
Accordingly
$$ E'^2 = \mathscr E^2 - \mathscr P^2~c^2 = (E + m~c^2)^2 - \left( \sqrt{ (E/c)^2 - (M~c)^2 } \right)^2~c^2,$$
which readily gives the expected expression for $E'$.
Best Answer
Both particles can be moving but often there is a stationary target particle at which "probing" particle are sent e.g. as in the Rutherford scattering experiment where alpha particles are fired at gold atoms.
"velocity of COM frame" is the velocity of the centre of mass of the system under consideration relative to the laboratory frame.
Relative to the centre of mass frame the total momentum of the system under consideration is zero.