It's a bad question. For one thing, answer (C) is utter nonsense. (Maybe that's a bit harsh. It might be just regular nonsense.) In order for something to convert gravitational potential energy into kinetic energy, it has to drop to a lower height under the influence of gravity. This does not happen during a collision. Collisions in physics are effectively instantaneous events; they occur at one point in space and time and then they're over and done with. There is no change in height by which GPE could be converted into KE during the collision. Whatever (kinetic) energy the balls run away with, they had to obtain it from the kinetic energy that the cart had coming into the collision.
Now, the kinetic energy of the cart at the point of the collision was converted from the gravitational potential energy that the cart had higher up the ramp. But that conversion was done by the cart alone; the balls had nothing to do with it.
The other reason I don't like this problem is that they don't tell you at which point on the ramp the cart has the speed of $5\text{ m/s}$. It's possible that the cart maintains a constant velocity as it goes down the incline, but that would require some mechanism to keep the cart from accelerating, and if some such mechanism is involved, it should be mentioned in the problem. If that is the case, the gravitational potential energy that the cart started out with would have been converted into some other form of energy, not kinetic. It might be heat, electricity, spring energy, etc. but there's no way to know unless they tell you what mechanism is keeping the cart from accelerating.
In a pinch, if you encountered this problem on the test and didn't have any opportunity to ask for clarification, I would just assume that $5\text{ m/s}$ is the speed at the end of the ramp, immediately prior to when the cart hits the balls. Why? The alternative is that the problem is unsolvable. If the speed of the cart coming into the collision is not $5\text{ m/s}$, you have no other information that would allow you to calculate what it is. (Self-check: do you understand why this is the case?)
Once you assume that the speed of the cart coming into the collision is $5\text{ m/s}$, you have a collision of 3 objects, each of which has a mass and initial and final velocities. All 3 masses, all 3 initial velocities, and two of the final velocities are known, so you should have enough information to solve for the third. If you don't find any solution, then the situation is impossible and the answer is (D); on the other hand, if you do find a solution for the final velocity of the cart, then that velocity will distinguish between choices (A) ($v_f = 0$), (B) ($v_f < 5\text{ m/s}$), and (C) ($v_f = 5\text{ m/s}$, if you ignore the stuff about energy being converted).
You are right - the net normal force was non-zero at some point during the collision. This is a consequence of Newton's Second Law.
Consider the acceleration of the clay/cart system as the clay is colliding with the cart. The clay has an upward acceleration, since it is being slowed down in the vertical direction by some force. Because of this vertical acceleration, the clay-cart system must also have a vertical acceleration during the collision. This means that the normal force during the collision will be higher than the normal force without any motion.
The standard concept of normal force = gravitational force only applies in situations where the system is not accelerating and $F = 0$. However, during the collision, the clay-cart system accelerates up. The situation is similar to the case of a scale in an elevator. As the elevator accelerates up or down, the overall force measured by the scale will change due to the acceleration of the system.
Consider the motion of the center of mass in the x-y plane before the collision, and this will be conserved afterwards. For example, if a very heavy ball of clay car was thrown with a high velocity directly into the path of a very slowly oncoming small toy car, the car's speed would likely increase by changing its direction. The center of mass of the original system was moving left rapidly, and so the center of mass of the clay-car system will also move left rapidly. To justify this with Newton's Laws, use the fact that the net force on each object in the collision is equal.
Best Answer
The cart and the ramp have to be able to move at different speeds from one another, otherwise the cart could never move up the ramp or back down it again. So they can't both have the same velocity at all times. However, the centre of mass of the system always keeps moving with the same velocity in the $x$ direction (both before and after the collision), and that's what you'll need in order to work out the answer. It's also useful to think about (i) what are the two $x$-velocities just at the instant of the collision, and (ii) what are the two $x$-velocities just at the very moment the cart reaches its highest point on the ramp?
This is a good question. The answer is that the gravitational force is acting vertically, whereas we're only considering momentum in the horizontal ($x$) direction. Momentum conservation applies separately to the $x$, $y$ and $z$ directions, so as long as there's no external force with a component in the $x$ direction, $x$-momentum is conserved.
It's not an easy question, but perhaps a useful hint is to think about it in terms of how much kinetic energy can get converted into potential energy. If you know $mgh$ then you can work out $h$.
I hope these hints help you to solve your problem...