I have a particle of rest mass $M$, total energy $E$ colliding with a stationery particle of rest mass $m$
The combined total energy $\mathscr E$ of this entire two-particle system (wrt. the "reference lab system" in which the particle of mass $m$ is stationary) is accordingly $$\mathscr E = E + m~c^2,$$
and the combined total momentum
$$\mathscr P = \sqrt{ (E/c)^2 - (M~c)^2 } + 0.$$
I have to show that [...] the total energy, $E'$ in the frame where their centre of mass is at rest is given by: [...]
For this I find helpful to think of the two-particle system described above as being replaced by just one (ficticious) particle of (suitable) mass $\mathscr M$, which carries the same total energy $\mathscr E$ and the same total momentum $\mathscr P$.
Obviously therefore $\mathscr M^2~c^4 = \mathscr E^2 - \mathscr P^2~c^2$.
Now, the point to note is that this (invariant) mass $\mathscr M$ is exactly the equivalent of the " total energy, $E'$ in the frame where their centre of mass is at rest" that you're seeking;
$$E' := \mathscr M~c^2.$$
Or in other words: the combined four-momentum vector of the two-particle system, as expressed in its center-of-mass frame (i.e. where its three-momentum vector is exactly zero) is equivalently
$ \{ \mathscr M~c, 0, 0, 0 \} = \{ E'/c, 0, 0, 0 \} $.
Accordingly
$$ E'^2 = \mathscr E^2 - \mathscr P^2~c^2 = (E + m~c^2)^2 - \left( \sqrt{ (E/c)^2 - (M~c)^2 } \right)^2~c^2,$$
which readily gives the expected expression for $E'$.
We do. The LHC accelerates two protons, each with 3.5 TeV of energy, giving a total of 7 TeV in the CoM frame (The energies are from the initial phase of the previous LHC run. Later in the run this was increased to 8 TeV and the combination of the two dataset was what discovered the Higgs boson. The energies are roughly doubling now for Run II, to 13 TeV).
The main reason for this is as you mentioned, the energy involved. In any frame, we have the following invariant quantity, $s = (p_1 + p_2)^2$ which its square root, $\sqrt{s}$, gives the Centre of Mass (CoM) energy for the experiment, and here $p_i$ represents the momentum four-vector for each particle i. In a collision where two particles are moving in opposite directions with equal energy we have the following:
$$ s \equiv (p_1 + p_2) \cdot (p_1 + p_2) = (E + E, \mathbf{p}_1 − \mathbf{p}_2 ) \cdot (E + E, \mathbf{p}_1 − \mathbf{p}_2 ) = (2E , 0 ) \cdot (2E, 0) = 4E^2$$
and now the CoM energy is given by the square root of this quantity, $$\to E_{CoM} = \sqrt{s} = 2E$$
In a an experiment where one of the particles is at rest (has mass $m_t$) and the other is travelling with momentum $\mathbf{p}$ (and has mass $m_b$) we have the following:
$$ s \equiv (p_1 + p_2) \cdot (p_1 + p_2) = (E_b + m_t, \mathbf{p}_b) \cdot (E_b + m_t, \mathbf{p}_b) = E_b^2 + m_t^2 + 2E_b m_t − p_b^2 = m_t^2 + m_b^2 + 2E_b m_t $$
Assuming the masses are negligible, we have the fixed target (FT) CoM energy,
$$\to E^{\text{FT}}_{CoM} = \sqrt{s} = \sqrt{2E_b m_t}$$
Thus we would need much more energy input in a fixed target experiment to achieve the same energies as in the case with two co-moving particles.
EDIT: Regarding a comment below which I think arises from confusion of what the CoM frame is. $\sqrt{s}$ gives the CoM energy in both cases. This is useful because we can now compare between a fixed target experiment and an experiment where both particles are accelerated at the same speed but in different directions.
So, say my collider has the capability to produce a magnetic field which at its maximum, can accelerate a charged particle so as to have energy of 3.5 TeV. Now in the case that we have two particles with the same energy going in opposite directions, we will give a total CoM energy of 7 Tev, following the result above. In the second case though, theres only one accelerating particle, hence $\sqrt{s} = \sqrt{2 \times 3.5 \times m_t}$ and since E $\ll$ m, this is always less than in the first case.
So be careful, because both experiments can be transformed into a CoM frame. In the CoM frame $|\mathbf{p_1}| = -|\mathbf{p_2}|$. Note this is true in both experiments, even in the second case where one of the particles is stationary. Well, the whole point is that we can use the above formulas so we can skip transforming to the CoM frame; we can compute this quantity directly.
Best Answer
In the expression $E^2 = (E_p+E_e)^2 - (E_e-pc)^2$, $E$ represents total energy in the zero momentum frame. The electron and proton do not annihilate so the energy available for particle creation can be found from the total energy minus the rest energies of the proton and electron. So the energy available for particle creation $E_x = E - m_pc^2 -m_ec^2$.