For a scenario such as this, can someone please explain why $\cos$ is used with the vertical and $\sin$ with the horizontal (with respect to the x-y coordinate system)? I'm used to $\sin$ being related to the $y$ coordinate or vertical direction and $\cos$ to the $x$ coordinate or horizontal direction and don't see what the reasoning is here.
[Physics] Sin and cos for normal forces on slanted plane
accelerationforcesfree-body-diagramgeometrynewtonian-mechanics
Related Solutions
The trouble is because you assumed that the final velocity of the small block is $\sqrt{2gh}$. This is true only if the wedge was stationary (in a frame of reference that is inertial), then what happens is that that the normal force from the wedge on the mass completely balances $mg\cos\alpha$, leaving the component $mg\sin\alpha$ down the wedge as you said.
But the situation is a little more complicated now, because the wedge is moving simultaneously as the small block slides down. So the forces don't balance out as described in the previous paragraph.
One can look at it in terms of energy to gain a better idea. The earth-wedge-mass system is isolated, so its total energy is conserved. The wedge doesn't gain or lose any potential energy, so the only change in potential energy comes from the mass. The change is $- mgh$. This must be distributed to the kinetic energies of BOTH the wedge and the mass. That is,
\begin{align} &\Delta K + \Delta U = \Delta E = 0 \nonumber \\ \implies & \Delta K_{wedge} + \Delta K_{block} - mgh = 0. \end{align}
if the wedge wasn't moving at all, we would then have $\Delta K_{wedge} = 0$, so \begin{align} \frac{1}{2}mv^2 = mgh \implies v = \sqrt{2gh} \end{align} like you said. But we see that if the wedge was moving, it 'eats' up some of the potential energy that would otherwise have gone to the mass. In other words, the small mass' speed will NOT be $v = \sqrt{2gh}$ at the bottom.
Having identified the flaw in your argument, how do we solve the question? There are a few ways. You can draw your force diagrams, carefully balancing out the forces and finding the geometric relation how the position of the mass relates to the position of the wedge. This analysis is perhaps easier in the wedge's frame of reference, but then you would have to add a fictitious force as it is not an inertial frame.
But the easiest analysis would be in terms of energy conservation, like the equation I gave you. We have \begin{align} \frac{1}{2}MV^2 + \frac{1}{2}mv^2 - mgh = 0. \end{align} Now all you have to do is find how $V$ is related to $v$. This is simple from conservation of momentum and some trigonometry, try it.
(Edit) I noticed after posting that you specifically highlighted the fact that $v = \sqrt{2gh}$ is with respect to the wedge. Lest you start pointing that out, this is not true, because the force the mass feels down the wedge is not $mg\sin\alpha$, because in this frame (wedge's frame, which is not inertial), there is the fictitious force.
Because $mg\cos x$ and $mg\sin x$ are orthogonal vectors, not colinear, and the norm of their sum is the Pythagorean sum. This kind of addition is pretty common in physics as well as in other places basic vector algebra shows up -- an elegant example is in statistics, where independent (orthogonal) random variables get added in a Pythagorean way, while multiple recordings of a variable get added up linearly.
The fact that we can do this is actually a very good indicator -- and part of the proof -- of the fact that physical forces are mathematical vectors, i.e. that it satisfies vector algebra. So the answer to the question is really an empirical one -- this is how forces get added, and can be mathematically explained as them being vectors.
Best Answer
By definition, sine of an angle for a right triangle is defined as the length of the side opposite the angle divided by the length of the hypotenuse. The cosine of that same angle is defined as the length of the side adjacent to the angle divided by the length of the hypotenuse.
In your drawing, for the angle shown, the indicated choice of trig functions is correct. The confusion that you are describing, which arises frequently in the high school physics classes that I teach, is in making the hidden assumption that the cosine of an angle is always the horizontal side of the associated triangle while the sine of the angle is always the vertical side of the associated triangle. This assumption is just plain false.