newtonian-mechanicswork
If the displacement is in the direction of the force, then work is positive.
If the displacement is in the opposite direction of the force, then work is negative.
If the displacement is perpendicular to the direction of the force then work is zero.
So, the work done depends on direction. So work should be a vector quantity. Then how is work a scalar quantity?
Lets place the stationary mass $M$ a the origin. In both cases the gravitational force is towards the origin, and is given by $$\vec{F}_{g}=-\frac{GMm}{r^{2}}\hat{r}$$
Thus in the first case, the work done by the gravitational force is $$W_{r\rightarrow\infty}=\int_{r}^{\infty}\vec{F}_{g}\cdot\vec{dr}=-\int_{r}^{\infty}\frac{GMm}{r^{2}}dr=-\frac{GMm}{r}$$
On the other hand, in the second case we just need to sweep the limits of integration $$W_{\infty\rightarrow r}=\int_{\infty}^{r}\vec{F}_{g}\cdot\vec{dr}=-\int_{\infty}^{r}\frac{GMm}{r^{2}}dr=\frac{GMm}{r}$$
which is exactly the opposite, and the signs are just what you've anticipated. An important thing to note is that also in both cases $$\vec{dr}=dr\hat{r}$$
and the sign is set by the integration limits. So in the first case $dr>0$ and in the second $dr<0$ (in a very informal manner). You ignored this fact when you wrote $\cos\left(180^{\circ}\right)$ in the dot product in the second case.
I have extensively updated my answer because the OP has the same conceptual problem as many of us more often in the context of deriving the electric potential due to a point charge and the gravitational potential due to a point mass.
My answer might be long winded but I thought it necessary because it has cause so many problems in the past and, no doubt, will do so in the future.
With the spring the change in elastic potential energy of a spring is equal to the work done by an external force whilst changing the length of the spring with the alternative definition the change in elastic potential energy of a spring is equal to the minus work done by the force exerted by the spring whilst changing the length of the spring.
Things go well when considering the spring increasing in length to some extension $x$ from its natural length $x=0$ as an external force is applied the result being that the change in the elastic potential energy of the spring is $\frac 12 k x^2$ where $k$ is the spring constant.
As expected $\frac 12 k x^2$ is a positive quantity ie the elastic potential energy of the spring has increased.
The OP has decided to look at the work done by spring when the length of the spring decreases and by using the fact that the force due to the spring is opposite in direction to the external force and the incremental displacement of the force $dx$ is such that it is also negative so that the dot product of the force and the incremental displacement is positive.
Doing the integration between the initial extension and the final extension leads to a negative value for the work done by the spring and hence an increase in the elastic potential energy of the spring which is obviously incorrect.
Let me try and explain the error in the second derivation by looking at a much simpler example in which we have a “magic” spring which exerts a constant force $F$ when its length varies between $x_{\rm A}$ and $x_{\rm B}$ with $x_{\rm B} > x_{\rm A}$.
To try and make it clearer as to what is happening I will also use a numerical example with the force being of magnitude $5\,\rm N$ and lengths $x_{\rm B}$ and $x_{\rm A}$ being $3\, \rm m$ and $1\, \rm m$ respectively.
It is a one dimensional problem and I define a unit vector in the positive x-direction $\hat x$ and the external force acting on the spring is $\vec F_{\rm external} = F_{\rm external} \hat x = +5 \hat x$.
Note that $F_{\rm external}$ is the component of the force in the $\hat x$ direction.
The position of the end of the spring changes from $\vec x_{\rm initial} = x_{\rm initial} \hat x$ to $\vec x_{\rm final} \hat x $ and the displacement of the end of the spring, which is the displacement of the external force, is $\Delta \vec x = \Delta x \hat x = \vec x_{\rm final} \hat x - \vec x_{\rm initial} \hat x$ which results in $\Delta x = x_{\rm final} - x_{\rm initial}$.
Again note that $ x_{\rm final}$ and $ x_{\rm initial}$ are components in the $\hat x$ direction.
The work done by the external force is $\vec F_{\rm external} \cdot \Delta \vec x = F_{\rm external} \hat x\cdot \Delta x \hat x= F_{\rm external}\, \Delta x = F_{\rm external} (x_{\rm final} - x_{\rm initial})$.
In the numerical example with the spring being stretched $F_{\rm external} = 5\, \rm N$, $x_{\rm initial} = 1 \, \rm m$ and $x_{\ rm final} = 3 \, \rm m$ so the work done by the external force is $5(3-1)= +10 \,\rm J$ and this is the increase in the elastic potential energy of the spring.
Note that we have evaluated $\displaystyle \int_ {x_{\rm initial}}^ {x_{\rm final}}F_{\rm external} \, dx$
But this formula also works when the length of the spring is decreasing.
This time you have $F_{\rm external} = 5\, \rm N$ (the external force direction is still the same), $x_{\rm initial} = 3 \, \rm m$ and $x_{\rm final} = 1 \, \rm m$ so the work done by the external force is $5(1-3)= -10 \,\rm J$ ie work is done on the external force and there is a decrease in the elastic potential energy of the spring.
Again we have evaluated $\displaystyle \int_ {x_{\rm initial}}^ {x_{\rm final}}F_{\rm external} \, dx$
At the risk of repetition, I would like to show how the analysis used above works if one considers the work done by the force exerted by the spring.
In this case $\vec F_{\rm spring} = F_{\rm spring} \hat x$ where $F_{\rm spring}$ is the component of force due to the spring $\vec F_{\rm spring}$ in the $\hat x$ direction.
The work done by the spring is $\vec F_{\rm spring} \cdot \Delta \vec x = F_{\rm spring}(x_{\rm final} - x_{\rm initial})$
In the numerical example $\vec F_{\rm spring} = -5 \hat x$ and and the work done by the spring force stretching the spring comes out to be $-10\, \rm J$ and the work done by the spring force when the length of the spring decreases is $-10\, \rm J$ which is what one would expect.
Now I will try and explain what was wrong with the method used by the OP in the case where the spring decreased in length.
The force due to the springs was in effect written as $- F\hat x$ in the knowledge that the direction of the force due to the spring was in the opposite direction to $\hat x$.
This immediately means that $F$ is positive.
The displacement was written as $–\Delta x \hat x$ on the assumption that the length of the spring would be decreasing.
If the minus sign is there what does that imply about $\Delta x$?
It must be positive!
The OP evaluated $-F \hat x \cdot -\Delta x \hat x = F \Delta x = F (x_{\rm final} – x_{\rm initial})$ using this integral $\displaystyle \int_ {x_{\rm initial}}^ {x_{\rm final}}F \, dx$ with $ x_{\rm final} < x_{\rm initial}$.
Using this integral with a constant force (for simplicity) produces a value for $\Delta x= x_{\rm final} – x_{\rm initial}$ which is negative when the spring is contracting.
What has been done is that by stating the incremental displacement as $-\Delta x \hat x$ then the implication is that $ x_{\rm final} > x_{\rm initial}$ and yet when the integral is set up $ x_{\rm final}$ was made smaller than $x_{\rm initial}$.
The resolution of this contradiction is fairly straightforward.
If you want to evaluate the correct integral when you use $-dx$ you must reverse the order of the limits which is equivalent to $\displaystyle \int_b^a f(x) (-dx) = \int_a^b f(x) dx$
Update as a clarification asked for by the OP
Still with $\hat x$ defining a direction I want to show that the work done is still $$\vec F \cdot \Delta\vec x = F \hat x \cdot \Delta x \hat x = F\Delta x = F (x_{\rm final} – x_{\rm initial})$$
This time considering the work done by the spring when it is compressed from $\vec x_{\rm initial} = -1 \hat x$ to $\vec x_{\rm final}=-3 \hat x$ with the spring exerting a force $\vec F_{\rm spring} = + 5 \hat x$
Work done by the spring
$$F\Delta x = F_{\rm {spring}} (x_{\rm final} – x_{\rm initial}) = = (+5)(-3-(-1)) = -10$$
The minus sign means that work has been done on the spring ie the elastic potential of the spring has increased.
In terms of an integral we have
$$\displaystyle \int ^{x_{\rm final}}_{x_{\rm initial}} F_{\rm spring} dx = \int ^{-3}_{-1} +5\, dx = -10$$
Further update as a clarification asked for by the @user35508
The work done by a spring is $\displaystyle \int ^{x_{\rm final}}_{x_{\rm initial}} \vec F_{\rm spring} \cdot d \vec x$ so what one needs is an expression for $\vec F_{\rm spring}$ in terms of the displacement of the end of the spring $\vec x = x \hat x$ and it is
$$\vec F_{\rm spring}= -kx \hat x$$
This is the correct form for the force exerted by the spring in that when the spring is stretched (x positive) it exerts a force in the $-\hat x$ direction and when the spring is compressed (x negative) it exerts a force in the $+\hat x$ direction.
$\text{Work done by spring} = \displaystyle \int ^{x_{\rm final}}_{x_{\rm initial}} \vec F_{\rm spring} \cdot d \vec x = \int ^{x_{\rm final}}_{x_{\rm initial}} -kx\hat x \cdot dx \hat x = \int ^{x_{\rm final}}_{x_{\rm initial}} -kx \, dx = \frac 12 k\left (x_{\rm initial}^2 -x_{\rm final}^2\right )$
For example if $x_{\rm initial} = -3$ and $x_{\rm final} = 0$ then this represents a compressed spring extending to its natural length and the work done by the spring is positive.
This was my original answer
Let the spring be the system and there be an external variable force $F \hat x$ acting on the spring in the direction $\hat x$.
The magnitude of the external force is $kx$ where $x$ is the extension of the spring.
This external variable force undergoes a displacement $dx \hat x$ and the work done by the external force is $F \hat x \cdot dx \hat x = F \, dx = kx \, dx$ and this is the change in the elastic potential energy stored in the spring.
Note that this equation is true when the spring is getting longer and also when the spring is getting shorter.
When the integration is done the limits of the integration define whether the spring is getting longer or shorter ie giving the correct sign to the incremental change in the length of the spring $dx$.
Work done by external force increasing the length of the spring from it natural length by $x$ is
$$\int ^x_0 kx \, dx = + \frac 12 kx^2$$
and this is the change in the elastic potential energy stored in the spring.
Now do the same again but have the spring start with extension $x$ and go back to its natural length.
Work done by external force is now
$$\int _x^0 kx \, dx = - \frac 12 kx^2$$
and this again is the change in the elastic potential energy stored in the spring.
If you consider the work done by the force exerted by the spring $-kx \hat x$ then this amount of work is equal to minus the change in the elastic potential energy stored in the spring.
Best Answer
From a mathematical point of view (ignoring integrals for now), we know that the work is defined to be
$$W = \vec{F}\cdot \vec{x}$$
By definition, the dot product of two vectors is a scalar. So that should be enough to convince you mathematically.
From a more intuitive point of view, remember that scalars can be negative or positive - this alone does not mean they are defining a direction. As you stated (correctly), the work does depend on the direction of the force. But this does not mean it is a vector itself (just look at the dot product above to convince yourself). Try to think of the force and displacement as more of a cause and effect type of a relationship though. You seem to be implying in your question that these entities are completely separate. If you push on an object in a certain direction, it is going to accelerate in that direction, unless some frictional force balances your push, so the net force is actually 0...then it will not accelerate at all.
If there is zero work, then will be no displacement (assuming 0 initial velocity). This is not because there is inherently no displacement...it is rather that there is no net force. Again, imagine pushing on a block downwards, so it does not move at all. This is because the normal force is pushing back at you, so there is no net force on the object at all. That is in fact why it does not move.
So depending on what direction you push on an object (and subsequently accelerates), you can define work to be positive or negative. This depends on your choice of coordinate system. But to reiterate - work will only be positive or negative. It will never have a direction associated with it.