There are two problems with the manipulations you've done.
First, the variables in equation (2) are ambiguously named. Equation (2) calculates the potential energy between a single mass $m$ and a mass distribution with total mass $M$. Then the equation should actually read
$$\Omega = - Gm \int \frac{dM}{R}.$$
If we instead write the differential as $dm$, it looks like $m$ is being integrated as well. This results in a meaningless extra factor of $1/2$ when the integration is performed.
Next, the integral over $dM$ shouldn't be naively performed as if $R$ is constant,
$$\int \frac{dM}{R} \neq \frac{M}{R}$$
in general. The issue is that every piece of mass $dM$ has its own radius $R$, so $R$ should be thought of as a function of $M$. If this doesn't make sense, just think about the discrete case,
$$\sum_i \frac{m_i}{R_i}$$
where a radius $R_i$ is associated with every bit of mass $m_i$.
In your particular case, where we're thinking about two point masses separated by a distance $R$, the quantity $R$ in the integrand really is constant, so we can pull it out for
$$\Omega = - \frac{Gm}{R} \int dM = -\frac{GMm}{R}$$
as expected. For a more general configuration, we would parametrize the masses and radii somehow to get a concrete integral, e.g. we could use the chain rule for
$$\int \frac{dM}{R} = \int \frac{dM}{dR} \frac{dR}{R}$$
where $dM/dR$ tells us the amount of mass in thin spherical shells of radius $R$. I explain how to do this kind of integral a bit more in this answer.
TL;DR There are few things wrong in how you understand the work-energy theorem:
- the work $W$ does not enter the energy diagram from both sides;
- if you include gravitational potential energy $U$ separately in the energy diagram, then you should not include it again as work $W$;
The work-energy theorem
The work-energy theorem says that the total work done on an object equals change in its kinetic energy
$$\Delta K = K_2 - K_1 = W \qquad \text{or} \qquad \boxed{K_1 + W = K_2}$$
where $W$ is total work done on an object. This means work done by all forces acting on the object! Note that by definition, work is a scalar value that can be either positive or negative.
One thing people usually forget is that the work done by gravitational and elastic (spring) forces equals negative change in the potential energy:
$$W_g = -\Delta U_g \qquad \text{and} \qquad W_e = -\Delta U_e$$
where $U_g = mgy$ is gravitational potential energy and $U_e = \frac{1}{2} k x^2$ is elastic potential energy. Since difference $\Delta$ always means final minus initial value, the work by gravitational and elastic forces is defined as
$$W_g = U_{g,1} - U_{g,2} \qquad \text{and} \qquad W_e = U_{e,1} - U_{e,2}$$
If we use this in the work-energy theorem we get
$$\boxed{K_1 + U_{g,1} + U_{e,1} + W_\text{other} = K_2 + U_{g,2} + U_{e,2}}$$
where $W_\text{other}$ is work done by forces other than gravitational and elastic forces.
Example: A freely-falling ball
As you have correctly identified, the change in kinetic energy can be negative, but the kinetic energy itself cannot. In example of dropping the ball from a height $h$, the energy diagram would be
$$\underbrace{\frac{1}{2} m 0^2}_{K_1} + \underbrace{mgh}_{U_{g,1}} = \underbrace{\frac{1}{2} m v_1^2}_{K_2} + \underbrace{mg0}_{U_{g,2}}$$
or in a general case
$$\frac{1}{2} m v_1^2 + mgy_1 = \frac{1}{2} m v_2^2 + mgy_2$$
which might be more familiar in this form
$$\boxed{v_2^2 = v_1^2 - 2g(y_2 - y_1)}$$
You must be careful what you use for $y_1$ and $y_2$ - subscript 1 denotes the starting point and 2 denotes the ending point.
Work by gravitational force equals negative $\Delta U_g$
As to why work done by gravitational force is negative change of potential energy, we start from the definition of work:
$$W = \vec{F} \cdot \vec{x} \qquad \text{or} \qquad W = \int \vec{F} \cdot d\vec{x}$$
where both force and displacement are vectors, which means they have both magnitude and direction. The work is defined as a scalar product of force and displacement, which means that only portion of force parallel to displacement does work, whereas perpendicular portion does no work. The scalar product finds that portion of work parallel to displacement. If force is constant and displacement is a straight line you can use the (simpler) left-hand side equation, but in all other cases you should use the right-hand side equation.
Let positive $\hat{\jmath}$ axis point upwards (away from Earth's center), then the gravitational force is
$$\vec{F}_g = -mg\hat{\jmath}$$
where $\hat{\jmath}$ is just a unit vector (magnitude equals to one) which defines direction of the force. The work done by gravitational force $\vec{F}_g$ over some distance $\Delta \vec{y}$ is
$$W_g = \vec{F}_g \cdot \Delta \vec{y} = -mg\hat{\jmath} \cdot (y_2 - y_1) \hat{\jmath} = -(mgy_2 - mgy_1) = -\Delta U_g$$
where $\hat{\jmath} \cdot \hat{\jmath} = 1$ is a scalar product between two unit vectors.
Best Answer
It looks like you figured out your mistake in the comments; here's the correct derivation. Let's start with your expression
$$\Delta E = \left[-\frac{GMm}{r}\right]^{e+h}_e = GMm \left(\frac{1}{e} - \frac{1}{e+h}\right),$$
which is the correct general expression for the change in gravitational potential energy outside a spherically symmetric body. If $h \ll e$, then we may expand
$$\frac{1}{e} - \frac{1}{e+h} = \frac{1}{e}\left[1-\frac{1}{1+h/e}\right] \approx \frac{1}{e}\left[1 - \left(1-\frac{h}{e} + \cdots\right)\right] = \frac{h}{e^2} + \cdots,$$
where we used the binomial expansion on $1/(1+h/e) = (1+h/e)^{-1} = 1-h/e + \cdots$, where the dots are terms that go like $(h/e)^2$. Plugging this back in, we get
$$\Delta E \approx \frac{GMmh}{e^2} = \frac{GM}{e^2} mh = mgh,$$
where we recognize that the gravitational acceleration at the surface of the Earth is $g = GM/e^2$. This is the expression you're looking for.
(For extra fun: if you keep the next term in the binomial expansion, i.e. the term that goes like $(h/e)^2$, you can show that if an oblong object like a rod is free to pivot about its center of mass, it will ever so slightly prefer to hang up-and-down rather than horizontally. This effect is partly responsible for the phenomenon of tidal locking, wherein, say, the rotation of the moon gets synced with that of the Earth, so that we only ever see the same side of the moon.)