forcesfree-body-diagramfrictionnewtonian-mechanicsstatics
There is only one force acting on the ladder which is its weight and it acts vertically downwards. Then why does the normal contact force from the vertical wall act horizontally on the ladder? There must be a horizontal force acting on the wall to exert a horizontal force on the ladder. What causes the horizontal force on the wall and what is it called?
A way of simplifying the problem is to combine the two weights using the idea of centre of gravity.
In your example the weight of the ladder $W$ positioned at a distance $L$ from the bottom of the ladder and the weight of the person positioned at an unknown distance from the bottom of the ladder $x$ can be combined to be a weight $4W$ at a position $y$ from the bottom of the ladder if $W\,L +3W\, x = 4W \, y$.
So you can now proceed having one weight $4W$ on the ladder and then find $y$ and hence find the required distance $x$.
However there is still a problem in that there is static friction at both ends of the ladder.
The simple way of solving such a problem is to assume that when the person gains maximum height the friction forces at both ends of the ladder are at a maximum given by $F = \mu \, N$ where $\mu$ is the coefficient of static friction and $N$ is the normal reaction.
Proving this to be the case is a little more difficult.
To do so let the resultant of the frictional force $\vec F$ and the normal reaction $\vec N$ be $\vec R$.
The frictional force can vary from a maximum value in one direction through zero to a maximum in the opposite direction so the resultant force $\vec R$ can point anywhere within the shaded area shown below.
The coefficient of static friction differs at each end of the ladder is different so the range of directions of the resultant forces (friction and normal reaction) at each differ as shown below with the shaded area showing the region where the lines of action the two resultant forces can intersect.
The problem has now been reduced to having three forces acting on the ladder.
A force on the ladder due to the ground $\vec R_{\rm g}$ , a force on the ladder due to the wall $\vec R_{\rm w}$ and a force on the ladder and person climbing it due to the gravitational attraction of the Earth $4\vec W$.
For these three forces to be in equilibrium their lines of action must intersect at a point.
Remembering that you are asked for the maximum height to which the person can climb the diagram below shows that if this condition is to be satisfied the static frictional force at both the ground and the wall must be a maximum.
Vertex $X$ of the quadrilateral $WXYZ$ is at the furthest point to the left for which there can be stable equilibrium and that represents the furthest to the left that the force $\vec 4W$ can act where the person is at the highest point on the ladder.
Update because of a comment by @AaronStevens
If the coefficient of friction at the bottom of the ladder gets smaller point $X$ in the diagram will move to the right.
The ladder alone cannot be in stable equilibrium if the coefficient of static friction at the bottom of the ladder is such that point $X$ is to the right of the centre of gravity of the ladder alone.
If there is no friction at the bottom of the ladder the ladder cannot be in stable equilibrium as there is no way of compensating for the horizontal component of force $\vec R_2$, the normal reaction at the top of the ladder.
It is possible for the ladder to be in stable equilibrium if there is no friction at the top of the ladder and friction at the bottom of the ladder.
The quadrilateral $WXYZ$ then degenerates into a horizontal straight line and if the left-hand end of that line is to the right of the centre of mass of the centre of mass of the ladder alone, stable equilibrium is not possible for the ladder alone.
So you need to look at each problem individually and rhere are some mechanics problems which you cannot solve without extra information.
For example if the line of action of the weight $4\vec W$ passed through the vertex $W$ of quadrilateral $WXYZ$ there is a range of values of $\vec R_1$ and $\vec R_2$ for which the ladder and person is in stable equilibrium,
It is a pity that the edX course 2.01 x Elements of Structures no longer exists even in archived form but you can still view a nice video about a Statically Indeterminate Problem which shows that for some mechanics problems one needs extra information to solve the problem.
When the beam is barely balanced, the plank nearly looses contact with the pillar labelled "A", which is why the normal force at "A" is zero. Any more weight at "B" will result in a torque imbalance, causing the plank to tip over.
To mathematically prove that a mass of $45\ \text{kg}$ at "B" is the maximum mass that results in a normal force of zero (at "A"), model the plank as a rigid body in static equilibrium.
These are the two equations that form the basis of the rigid object in static equilibrium model: $$ \sum \vec{F} = 0 \ \ \text{and} \sum \vec{\tau_z} = 0$$ The sum of torques can be made about any axis.
Call the normal forces at "A" and "C" as $\vec{n_A}$ and $\vec{n_C}$ respectively. Let the plank's mass be $M = 15 \ \text{kg}$, and the boy's weight be $\vec{F_g}$ = $(45 \text{kg}).\vec{g}$
The net torque about an axis perpendicular to point "A" is zero, so: $$\sum \tau_A = 0 \rightarrow \ -Mg(\overline{\text{AO}}) \ + n_C(\overline{\text{AC}}) -F_g(\overline{\text{AB}}) = 0$$
Solving this equation should yield : $n_C = 588N$
The net force on the plank is also zero, so: $$\sum F = 0 \rightarrow Mg \ + F_g - n_C \ -n_A = 0$$ which yields $\boxed{n_A = 0}$
As an exercise, try using this method to caluclate $n_A$ if the mass of the boy was $46 \ \text{kg}$. You will get a negative normal force at "A" which is unphysical.
Hope this helps.
Best Answer
Think about how a ladder stands up in real life. Would the ladder stay in the orientation shown in your image if the ground were ice? No! The reason? Friction.
The friction force, represented by $\vec{F}_{ff}$ in the figure acts to prevent the ladder from sliding to the right.
There are actually 5 forces acting on this ladder: