It's not correct that runners lean forward to begin a race in order to increase friction. They lean forward because otherwise, they would experience no propulsion whatsoever because static friction is zero when the runner is completely upright.
When the runner leans forward and flexes his leg muscles, he exerts a horizontal force on the track in the backward direction. The track responds by exerting an equal and opposite frictional force (unless there is slipping) on the runner in the forward direction that propels him forward.
Generally speaking, the more a runner leans forward at the start, the larger the horizontal component of the force exerted by his legs against the ground, and the larger the frictional force he will experience. As a result, his initial acceleration will be greater.
Part A
$\mu$ large implies that the wedge does not move. Therefore the two forces on the small masses along their direction of movement are:
$$F_1 = m_1 g, \quad F_2 = m_2 g \sin(\alpha).$$
They pull in different directions on the pulley, so the net acceleration (with respect to $m_1$ going down) is:
$$F = F_1 - F_2 = g [m_1 - m_2 \sin(\alpha)].$$
Then the acceleration of both masses is:
$$ a = \frac Fm = \frac{g}{m_1 + m_2}[m_1 - m_2 \sin(\alpha)].$$
You should not be able to introduce a minus error just by choosing a convention. I assume that you just did not flip all the minus signs.
Part B
The wedge has to be at rest, so the force of friction has to compensate the tangential forces exactly. This means
$$ \mu F_\text{down} \overset != F_\text{tangential}. $$
I now construct the forces that act on the masses:
The tensions will change with the acceleration:
$$ T_1 = m_1[g-a], \quad T_2 = m_2[g\sin(\alpha) + a].$$
To digest this, set $a =0$, the static case. Then the tensions are just the forces that gravity exerts on the masses. The extreme case, where $m_1$ is free falling is $a = g$. Then $T_1$ has to be zero, since the object is free falling.
The mass $m_2$ exerts a normal force onto the wedge that will push it to the left. The tension $T_2$ will drag on the pulley which will then push on the wedge to the right. The force to the left is given by
$$ m_2 g \cos(\alpha)\sin(\alpha)$$
whereas the force to the right (just the $x$-component) is given by
$$ m_2[g \sin(\alpha) + a] \cos(\alpha).$$
In the static case $a = 0$, those two forces are equal and do not make the wedge move. In the dynamic case, they try to move the wedge. This is what you meant by conservation of linear momentum, just written down with the forces.
The net force to the right (assuming $a > 0$ then is
$$ m_2 a \cos(\alpha) = F_\text{tangential}.$$
Now the forces that push the wedge downward and give it a frictional force. The mass $M$ will drag downward. Then $T_1$ will push the pulley down, so does the $y$-component of $T_2$. The last thing is the $y$-component of the normal force. The last two mostly add up to just $m_2 g$, there is still a $m_2 a \sin(\alpha)$ left from the dynamics in $T_2$:
$$\underbrace{m_1[g-a]}_{T_1} + \underbrace{m_2[g\sin(\alpha) + a]\sin(\alpha)}_{{T_2}_y} + m_2 g \cos(\alpha)^2$$
Together, this is:
$$F_\text{down} = Mg + m_1[g-a] + m_2[g + a\sin(\alpha)].$$
Now set them equal, plug in $a$ and solve for $\mu$. You might want to omit all the terms with $a$ here, since that makes it really complicated. If $M$ is large, you can do that without a large error.
Best Answer
A way of simplifying the problem is to combine the two weights using the idea of centre of gravity.
In your example the weight of the ladder $W$ positioned at a distance $L$ from the bottom of the ladder and the weight of the person positioned at an unknown distance from the bottom of the ladder $x$ can be combined to be a weight $4W$ at a position $y$ from the bottom of the ladder if $W\,L +3W\, x = 4W \, y$.
So you can now proceed having one weight $4W$ on the ladder and then find $y$ and hence find the required distance $x$.
However there is still a problem in that there is static friction at both ends of the ladder.
The simple way of solving such a problem is to assume that when the person gains maximum height the friction forces at both ends of the ladder are at a maximum given by $F = \mu \, N$ where $\mu$ is the coefficient of static friction and $N$ is the normal reaction.
Proving this to be the case is a little more difficult.
To do so let the resultant of the frictional force $\vec F$ and the normal reaction $\vec N$ be $\vec R$.
The frictional force can vary from a maximum value in one direction through zero to a maximum in the opposite direction so the resultant force $\vec R$ can point anywhere within the shaded area shown below.
The coefficient of static friction differs at each end of the ladder is different so the range of directions of the resultant forces (friction and normal reaction) at each differ as shown below with the shaded area showing the region where the lines of action the two resultant forces can intersect.
The problem has now been reduced to having three forces acting on the ladder.
A force on the ladder due to the ground $\vec R_{\rm g}$ , a force on the ladder due to the wall $\vec R_{\rm w}$ and a force on the ladder and person climbing it due to the gravitational attraction of the Earth $4\vec W$.
For these three forces to be in equilibrium their lines of action must intersect at a point.
Remembering that you are asked for the maximum height to which the person can climb the diagram below shows that if this condition is to be satisfied the static frictional force at both the ground and the wall must be a maximum.
Vertex $X$ of the quadrilateral $WXYZ$ is at the furthest point to the left for which there can be stable equilibrium and that represents the furthest to the left that the force $\vec 4W$ can act where the person is at the highest point on the ladder.
Update because of a comment by @AaronStevens
If the coefficient of friction at the bottom of the ladder gets smaller point $X$ in the diagram will move to the right.
The ladder alone cannot be in stable equilibrium if the coefficient of static friction at the bottom of the ladder is such that point $X$ is to the right of the centre of gravity of the ladder alone.
If there is no friction at the bottom of the ladder the ladder cannot be in stable equilibrium as there is no way of compensating for the horizontal component of force $\vec R_2$, the normal reaction at the top of the ladder.
It is possible for the ladder to be in stable equilibrium if there is no friction at the top of the ladder and friction at the bottom of the ladder.
The quadrilateral $WXYZ$ then degenerates into a horizontal straight line and if the left-hand end of that line is to the right of the centre of mass of the centre of mass of the ladder alone, stable equilibrium is not possible for the ladder alone.
So you need to look at each problem individually and rhere are some mechanics problems which you cannot solve without extra information.
For example if the line of action of the weight $4\vec W$ passed through the vertex $W$ of quadrilateral $WXYZ$ there is a range of values of $\vec R_1$ and $\vec R_2$ for which the ladder and person is in stable equilibrium,
It is a pity that the edX course 2.01 x Elements of Structures no longer exists even in archived form but you can still view a nice video about a Statically Indeterminate Problem which shows that for some mechanics problems one needs extra information to solve the problem.