It's a bad question. For one thing, answer (C) is utter nonsense. (Maybe that's a bit harsh. It might be just regular nonsense.) In order for something to convert gravitational potential energy into kinetic energy, it has to drop to a lower height under the influence of gravity. This does not happen during a collision. Collisions in physics are effectively instantaneous events; they occur at one point in space and time and then they're over and done with. There is no change in height by which GPE could be converted into KE during the collision. Whatever (kinetic) energy the balls run away with, they had to obtain it from the kinetic energy that the cart had coming into the collision.
Now, the kinetic energy of the cart at the point of the collision was converted from the gravitational potential energy that the cart had higher up the ramp. But that conversion was done by the cart alone; the balls had nothing to do with it.
The other reason I don't like this problem is that they don't tell you at which point on the ramp the cart has the speed of $5\text{ m/s}$. It's possible that the cart maintains a constant velocity as it goes down the incline, but that would require some mechanism to keep the cart from accelerating, and if some such mechanism is involved, it should be mentioned in the problem. If that is the case, the gravitational potential energy that the cart started out with would have been converted into some other form of energy, not kinetic. It might be heat, electricity, spring energy, etc. but there's no way to know unless they tell you what mechanism is keeping the cart from accelerating.
In a pinch, if you encountered this problem on the test and didn't have any opportunity to ask for clarification, I would just assume that $5\text{ m/s}$ is the speed at the end of the ramp, immediately prior to when the cart hits the balls. Why? The alternative is that the problem is unsolvable. If the speed of the cart coming into the collision is not $5\text{ m/s}$, you have no other information that would allow you to calculate what it is. (Self-check: do you understand why this is the case?)
Once you assume that the speed of the cart coming into the collision is $5\text{ m/s}$, you have a collision of 3 objects, each of which has a mass and initial and final velocities. All 3 masses, all 3 initial velocities, and two of the final velocities are known, so you should have enough information to solve for the third. If you don't find any solution, then the situation is impossible and the answer is (D); on the other hand, if you do find a solution for the final velocity of the cart, then that velocity will distinguish between choices (A) ($v_f = 0$), (B) ($v_f < 5\text{ m/s}$), and (C) ($v_f = 5\text{ m/s}$, if you ignore the stuff about energy being converted).
$p_{photon} = \frac{h}{\lambda}$, with its energy $E_{photon} = pc$
The momentum of an electron is like any other object. Remember to account for relativistic effects!
You now have two equations and two undetermined variables, for example:
$a +b = 1$ and $2a - b = 0$
One way to solve this is to rearrange the first to $a = 1-b$ and insert that into equation 2, solve for b and then find a with either initial equation.
You've botched the vectors in your momentum equation. Pick to the right as positive and to the left as negative. You'll find that pre-collision $p_{photon}$ is positive and $p_{electron}$ is negative and viceversa after the collision.
Try to avoid using eV (and other non-SI units) if you can. I find it's just another way for errors to creep into your calculation.
For one, your expression of relativistic kinetic energy is wrong.
Best Answer
If you do the calculation completely in the CM system, you must transform back to the lab to answer the question. Kinetic energy is depends on the frame of reference.
An easier direction to proceed is to use a Lorentz-invariant quantity:
$$\left[E_{lab}^2-p_{lab}^2c^2\right]_{before} = \left[E_{CM}^2-p_{CM}^2c^2\right]_{after}.$$
Of course, $p_{CM}=0$, and $E_{CM}$ is simply the total mass-energy. The real work comes with determining the total lab energy and total lab momentum before the collision. That's not tremendously hard, but it does require some thought. Two massive particles, one at rest and the other moving.