Find the velocity of the triangular block when the small block reaches the bottom:
Here is what I did:
The final velocity(at the bottom)of the small block of mass m is $\sqrt{2gh}$ along the plane of the incline with respect to the triangle (due to uniform acceleration $g\sin a$ covering distance $\frac{h}{\sin a}$). Let the velocity of the triangular wedge be $V$. Since there is a net external force in the vertical direction, linear momentum is conserved only in the horizontal direction.
Then,the velocity of the small block with respect to ground is $$ \Bigl(\sqrt{2gh} \cos a\ + V \Bigr) ,$$ we are not considering the direction of $V$, which intuitively should be leftward, but we take rightward. Afterward we should get a negative sign indicating the left direction.
Applying conservation of linear momentum in the horizontal direction we get
$$ MV + m \Bigl(\sqrt(2gh) \cos a\ + V \Bigr) = 0 .$$
Thus we find that $$ V = \frac{-m \Bigl(\sqrt(2gh) \cos a\ \Bigr)}{m+M}. $$
However, my book mentions that the answer is something different. I wouldn't like to mention it here because I do not want reverse-engineering from the answer. Please help and explain where I may be wrong.
Best Answer
The trouble is because you assumed that the final velocity of the small block is $\sqrt{2gh}$. This is true only if the wedge was stationary (in a frame of reference that is inertial), then what happens is that that the normal force from the wedge on the mass completely balances $mg\cos\alpha$, leaving the component $mg\sin\alpha$ down the wedge as you said.
But the situation is a little more complicated now, because the wedge is moving simultaneously as the small block slides down. So the forces don't balance out as described in the previous paragraph.
One can look at it in terms of energy to gain a better idea. The earth-wedge-mass system is isolated, so its total energy is conserved. The wedge doesn't gain or lose any potential energy, so the only change in potential energy comes from the mass. The change is $- mgh$. This must be distributed to the kinetic energies of BOTH the wedge and the mass. That is,
\begin{align} &\Delta K + \Delta U = \Delta E = 0 \nonumber \\ \implies & \Delta K_{wedge} + \Delta K_{block} - mgh = 0. \end{align}
if the wedge wasn't moving at all, we would then have $\Delta K_{wedge} = 0$, so \begin{align} \frac{1}{2}mv^2 = mgh \implies v = \sqrt{2gh} \end{align} like you said. But we see that if the wedge was moving, it 'eats' up some of the potential energy that would otherwise have gone to the mass. In other words, the small mass' speed will NOT be $v = \sqrt{2gh}$ at the bottom.
Having identified the flaw in your argument, how do we solve the question? There are a few ways. You can draw your force diagrams, carefully balancing out the forces and finding the geometric relation how the position of the mass relates to the position of the wedge. This analysis is perhaps easier in the wedge's frame of reference, but then you would have to add a fictitious force as it is not an inertial frame.
But the easiest analysis would be in terms of energy conservation, like the equation I gave you. We have \begin{align} \frac{1}{2}MV^2 + \frac{1}{2}mv^2 - mgh = 0. \end{align} Now all you have to do is find how $V$ is related to $v$. This is simple from conservation of momentum and some trigonometry, try it.
(Edit) I noticed after posting that you specifically highlighted the fact that $v = \sqrt{2gh}$ is with respect to the wedge. Lest you start pointing that out, this is not true, because the force the mass feels down the wedge is not $mg\sin\alpha$, because in this frame (wedge's frame, which is not inertial), there is the fictitious force.