Why is it that internal forces do not change the potential energy of the system?
Assume that a man stands on earth with a dumbell. If the dumbell is the system, when it is raised a height h, the work done by the man is +mgh and the work done by the earth is -mgh.
If the system is the earth and the dumbell, then the man will increase the potential energy of the system by +mgh. Where does the gravitational force of the earth on the dumbell go?
Best Answer
Internal forces are the only contributors to potential energy.
Potential energy is the energy associated with the configuration (relative positions) of a collection objects. The potential energy of a single point particle is not defined. As the configuration of the system changes, its potential energy changes according to the definition $\Delta{\mathrm{(P.E.)}} = -W_\mathrm{internal}$ where $W_\mathrm{internal}$ is the work done internally, that is, by the various internal forces against the internal components of the system.
In order to define potential energy, you need to have internal components applying forces to one another, and the components need to be able to move relative to one another. You need at least two objects.
Consider the system consisting of only the dumbbell. You have implicitly modeled the dumbbell as a point particle. That is, the only attributes of interest to your analysis are its position and its mass. The internal structure, which would be of interest in other questions, say, concerning temperature, is ignored. You have applied two external forces, gravity, and the contact force of your hand. Not only is there no change in potential energy, there is no potential energy at all. Potential energy does not exist in a system consisting of, or modeled by, a single point particle. I'm not saying that the P.E. is zero. I'm saying that it is not defined.
Consider the system consisting of the dumbbell and the earth. Now I have two objects modeled as point particles, one external force: the force of your hand, and one internal force: the mutual gravitational attraction between the Earth and the dumbbell. The internal work done in lifting the dumbbell (that is, increasing the distance between them) is $W_\mathrm{internal} = -mgh$. The minus sign comes from the fact that the displacement is in the direction opposite to the direction of the force. (Gravity down, displacement up.) so the change in potential energy of the system is $$\Delta\mathrm{(P.E.)} = -W_\mathrm{internal} = -(-mgh) = mgh$$ The external force of your hand on the dumbbell plays no role at all.
One of the roots of your misunderstanding is the unfortunate choice made by almost all introductory textbooks in introducing the subject of potential energy by looking at raising and lowering objects in a static gravitational field (at the surface of the earth). I know of a particular textbook that had a wonderful and correct description of potential energy. Until the next edition came out, when that description disappeared, and the conventional description appeared in its place.
addendum
There's something that needs clarification. In what I've written above, I modeled all of the objects as point particles having no internal structure. That includes my model of the composite system: I took that as a point particle also, this one located at the center of mass of the composite system. External forces can accelerate the system, but that's all it can do. $F_\mathrm{net} = ma_\mathrm{cm}$, and the kinetic energy of the system increases in accordance with the usual analysis: $\Delta KE = \vec{F_\mathrm{net}}\cdot \Delta \vec{x}_\mathrm{cm}$
If I take a better model for the composite system I might find that it is deformable. Suppose the system is a rubber ball. Now when I apply the external force, the object will deform. The center of mass still obeys $F_\mathrm{net} = ma_\mathrm{cm}$ and we still have $\Delta KE = \vec{F_\mathrm{net}}\cdot \Delta \vec{x}_\mathrm{cm}$. But notice that since the object is deforming, the displacement of the center of mass is not the same as the displacement of the point of application of the force. The kinetic energy of the system increases as before: $\Delta KE = \vec{F_\mathrm{net}}\cdot \Delta \vec{x}_\mathrm{cm}$, but the point of application of the force moves farther than $\Delta \vec{x}_\mathrm{cm}$. More work was done than was needed to accelerate the system. Some of this extra work ends up as internal potential energy. Some as internal kinetic energy, some as thermal energy $\ldots$. None of these energies exist in the point particle model of the system, but in a real system they do exist. In that sense, external forces can change the potential energy of a system.
But even in that more realistic analysis, the change in the potential energy of the system is due to the internal forces, and internal work. There is an external influence that causes deformation, but the change in potential energy is calculated from internal work.