I think you just forgot that the $\int_A^B F\,dl$ is not a scalar expression. Rather it should be written in a form $\int_A^B \vec{F}\cdot d\vec{l}$. Then it comes to the sign of the scalar product:
$$\vec{F}\cdot d\vec{l}=F\,dl\,\cos\theta$$
where the angle $\theta$ is taken between the vector $\vec{F}$ and the direction of the tangent to the integration path from $A$ to $B$. Then, in your first example,
$W_{A \to B}=\int_{r_A}^{r_B} F(r) dr = \int_{r_A}^{r_B} \left(-\frac{GMm}{r^2} \right)dr$
the path could go with any slope, but the gravity is always directed downwards, along the $r$ axis. That means, we can always take $(\pi-\theta)$ as the angle between the vector $d\vec{l}$ and the $r$ axis, that is
$$dl\,\cos(\pi-\theta)=dr$$
but $\cos(\pi-\theta)=-\cos\theta$ and thus we have
$$\vec{F}\cdot d\vec{l}=-F\,dr=-\frac{GMm}{r^2}dr$$
For your second example:
...we also should change the sign, because the gravitational force is always a force of attraction.
what the authors actually mean is that: the Coulomb's and Newton's forces have exactly the same expressions, but the sign conventions for them are different. The Newton's force is defined that if all the quantities ($M$, $m$ and $r$) are positive, then the vector of the force is directed towards the other body. But for the Coulomb's force, if all the quantities ($q_1$, $q_2$ and $r$) are positive, then the vector of the force is directed away from the other charge. That becomes manifest if we take the vector expressions for these forces:
$$\vec{F}_N=-\frac{GMm\,\vec{r}}{r^3}\qquad\vec{F}_C=\frac{q_1q_2\,\vec{r}}{4\pi\epsilon_0\,r^3}$$
Now the different signs are clearly seen.
"...from point $A$ to point $B$..." - ...as I understand it - the work that I must do is always $U_B-U_A$. However the work that the force that is being created by the field do_es_ is always $U_A-U_B$, am I correct?
Yes this is correct.
The mnemonic rule is very simple: $U$ is like the height of the slope. When you go up, $U_B>U_A$, and it is you who does the work. But when you go down, $U_A>U_B$, and it is the field force who does the work.
The value of the potential energy is totally arbitrary until you define a position where the potential energy is zero when in effect you are dealing with differences in potential energy.
$$V_g = -mgz$$ where $m$ is the mass, $g$ is the gravity and $z$ is
the positive (?) height from the zero potential energy.
Your confusion stems from this statement because it lacks detail.
In the diagram below I have defined a positive $z$ direction.
With the direction of $z$ increasing defined if you move a mass $m$ upwards by a distance $z$ then the change in potential energy is $(-mg)(-z) = mgz$.
This comes from the fact that an upward external force (-mg) has moved the mass upwards a distance (-z).
As expected it is an increase in potential energy but note that going up means that z becomes more negative and that is why it is minus $z$ in the evaluation of the change in potential energy.
If you want an actual value of potential energy then you must define a zero of potential energy.
If you had chosen position $C$ as your zero of potential energy then after lifting the mass up to position $A$ from position $C$ the potential energy of the mass is $(-mg)(-L) = mgL$.
On the other hand if you had chosen the zero of potential energy at position $A$ the potential energy of the mass at position $C$, noting that to go from position $A$ to position $C$ the mass is moving in the positive z-direction, is $(-mg)(+L) = -mgL$.
The change in potential energy in going from position $C$ to position $A$ is the same irrespective of where the zero of potential energy was defined.
final potential energy at $A$ - initial potential energy at $C$ is equal to
$mgL - 0 = 0 -(-mgL) = mgL$
You could have chosen the positive z-direction as upwards and used as the change in potential energy $mgz$ noting that to lift the mass up the external force is in the positive z-direction.
If you had chosen position $C$ as your zero of potential energy then after lifting the mass up to position $A$ from position $C$, the potential energy of the mass is $(+mg)(+L) = mgL$, the same as before.
On the other hand if you had chosen the zero of potential energy at position $A$ the potential energy of the mass at position $C$ is $(+mg)(-L) = -mgL$, again the same as before.
So the advice is, define a positive direction and if you want to find the potential energy define a position where the potential energy is zero.
Best Answer
In general, potential energy is only well-defined up to an additive constant. The physically relevant quantity is the difference in potential energy between two points. So there is nothing wrong with some points having a negative potential energy. For any given problem, you'll define your reference point to make your equations as simple as possible.
In this particular example, it looks like the reference point is defined as the base of the inverted pendulum, where it attaches to the car (or ground, if the car isn't present in your version of the problem). $h$ is then the height above the car (ground). In this case, it's probably easiest to measure the height of the pendulum's bob in terms of that definition. But you could instead define your reference point to be the point of highest reach of the pendulum's bob. $h$ would then be measured as height above that point. Since all the points you're interested are below that point, $h$ would be negative for all points, and potential energy would always be negative. And that's okay; there's nothing inherently wrong with that. It just might be a bit harder to work with the resulting equations.
This is beyond the scope of this particular example, but $PE = mgh$ only works for systems near the surface of the earth, where $h$ is much smaller than the radius of the earth. If you have to work with distances that are similar in size to the radius of the earth, you have to use the full (Newtonian) form $PE = -GMm/r$, where I'm using $r$ to mean the distance between the object and the center of the earth. In cases where you have to use this form, the easiest reference is usually to define the potential energy to be zero when the object is infinitely far from the earth. In this case, all points a finite distance from the earth have a negative potential energy.