It appears you made a few mistakes.
The formula $E_P = mgh$ is only an approximation for objects near the ground. The more complete formula is
$ E_P = -\frac{\mu m}{r} $
where $\mu = 398600.44$ is Earth's standard gravitational parameter, and $r$ is the distance between the object and the Earth's center of gravity.
Especially note the negative sign; this has to do with the definition of potential energy in the context of orbits. This is where I think you went wrong.
Also, where did you find $V = 11.068$ km/s for a geostationary orbit? That looks more like an escape speed than a normal orbital speed...Indeed, if you look up the altitude for a geostationary orbit you see that it is $35768$ km above the equator. That means the total pathlength traversed by the satellite in one stellar day is
$2\pi \cdot ( 35768+R_E) \approx 264,811 $km
making the speed
$264,811 \mathrm{\ km} / 86164 \mathrm{\ seconds} \approx 3.07 \mathrm{\ km/s}$
so much much slower than the ~11 km/s you stated. Lumping all this together:
$E_P^{GEO} \approx -\frac{398600.44}{42164} = -9.45 $ kJ/kg
$E_K^{GEO} \approx \frac{3.07^2}{2} = 4.71 $ kJ/kg
$E_{tot}^{GEO} = 4.71-9.45 = -4.74 $ kJ/kg
while for the other orbit
$E_P^{alt} \approx -\frac{398600.44}{45000} = -8.86 $ kJ/kg
$E_K^{alt} \approx \frac{2.98^2}{2} = 4.44 $ kJ/kg
$E_{tot}^{alt} = 4.44-8.86 = -4.42$ kJ/kg
which is indeed higher than the GEO orbit.
This makes sense -- you need to input a lot more energy to let anything escape from Earth's gravity than, say, an apple falling to the ground (which is also in an "orbit", albeit one far closer to the Earth, and not exactly on an escape trajectory).
If what you say would be true, everything would simply fall up and escape the Earth. There are a few experiments that will show that that is not actually what happens :)
With regard to your statement about the moon: the moon is indeed slowly escaping from the Earth. The mechanism here is that the Moon is gaining orbital speed at the expense of Earth's rotational momentum, through tidal interaction.
Roughly translated: as Earth's rotation slows down, the Moon speeds up, making the Moon progress farther away from the Earth, towards a lower speed.
The total energy in that higher orbit is higher, because the drop in speed is disproportionally small in relation to the gain in potential energy. Eventually, after a few million years of repeating the above, the moon will have gained enough energy to escape the Earth and orbit the Sun on its own.
It helps if you consider the components of the acceleration of the smaller planets due to the gravitation force of each other planet. Here is a rough diagram showing the components of acceleration for each planet, assuming the largest does not accelerate due to its large mass:
The red arrow shows the component of acceleration of a planet due to the gravity of the largest planet. The green arrow is the component due to the other planet (not the largest).
Now, consider the bodies accelerate only by the red components of acceleration (i.e. ignore the gravitation effects between the two smaller planets). As the centres of gravity for the smaller planets are the same distance away, the motion of the two smaller planets will be perfectly symmetrical, and both planets hit the largest at the same time (assume smaller planets have similar radii/size).
Now, let us add the effects of the green arrows (i.e. the gravitation effect between the smaller planets). The angle between the red arrow and the green arrow is less than 90°, so this means that the green arrow will add to the acceleration of the planet towards to largest planet. Let us assume that the green arrows are sufficiently small so that the total component of acceleration perpendicular to the line connecting each small planet to the largest planet does not cause large rotational effects about the largest planet (i.e. effects such that the small planets do not travel in approx. straight lines towards the largest planet, complicating matters, and possibly throwing the smaller planets into orbit around the larger!).
The less massive planet will have a larger green arrow, and the larger of the two green arrows has a greater contribution to the component of acceleration in the radial direction (radial, as in the line connecting the smaller planet to the biggest planet). This means that, in this instance, the less massive planet is accelerating slightly faster than the slightly more massive planet towards the biggest planet. From this, it is sufficient to assume that... drumroll the smaller planet will hit the largest planet first!
This is true for all values of d and h, assuming the values don't cause the smaller planets to collide with each other first, and that the path between each smaller planet, and the largest planet is approximately straight. (i.e. d is above a critical value)
Best Answer
A small portion of any smooth curve looks the same as a small piece of a parabola in the limit. Choose a coordinate system so that the tangential direction in the middle of the segment is along the $x$ axis and choose a translation for the middle of the segment to sit at $(0,0,0)$, the origin of the coordinates. Then $y,z$ on the curve (ellipse etc.) may be viewed as functions of $x$ and these functions $y(x),z(x)$ may be Taylor-expanded. The first nontrivial term is $$ y(x) = a_2^y x^2 +O(x^3)$$ because the absolute and linear terms were made to vanish by the choice of the coordinates. But neglecting the $x^3$ and other pieces, this is just an equation for a parabola.
(A similar comment would apply to $z(x)$ and one could actually rotate the coordinates in the $yz$ plane so that $a_2^z$ equals zero.)
So a "supershort piece" of a curve is always a straight line. With a better approximation, a "very short" piece is a parabola, and one may refine the acceptably accurate formulae by increasingly good approximations, by adding one power after another.
But near the perigeum (the closest approach to the source of the gravitational field), we may actually describe a limiting procedure for which the "whole" ellipse – and not just an infinitesimal piece – becomes a parabola. As the maximum speed of the satellite (the speed at the perigeum etc.) approaches the escape speed, while the place of the perigeum is kept at the same point, the elliptical orbit approaches a parabolic trajectory – the whole one. Note that in this limiting procedure, the furthest point from the center of gravity goes to infinite distance and the eccentricity diverges, too.
So the parabola is a limit of a class of ellipses. In the same way, a parabola is a limiting case of hyperbolae, too. In fact, in the space of conics, a parabola is always the "very rare, measure-zero" crossing point from an ellipse to a hyperbola. This shouldn't be shocking. Just consider an equation in the 2D plane $$ y-x^2-c=ay^2 $$ which is a function of $a$. For a positive $a$, you get an ellipse; for a negative $a$, you get a hyperbola. The intermediate, special case is $a=0$ which is a parabola. One may also parameterize these curves as conic sections. The type of the curve we get will depend on the angle by which the plane is tilted relatively to the cone; the parabola is again the transition from ellipses at low angle to hyperbolae at a high angle.