There are lots of questions here that I will try to answer, hopefully I'll get to them all...
Creature Comforts
It's hard to "just fly higher" when you consider passenger planes. Supersonic military aircraft like the SR-71 do fly ridiculously high. It's service ceiling is 85,000 feet! But, it has the advantage that it doesn't need to keep anybody but the pilot comfortable. The issue deals with pressurization. As you increase altitude, the aircraft must also be able to withstand a larger pressure differential if the cabin will be kept at a comfortable pressure. Most very high altitude military aircraft do not pressurize the cabin; rather, the pilot wears a pressure suit. Imagine if you had to suit up for a flight to visit relatives!
It's not that we can't build a plane that can withstand the pressure difference, but doing so would require very heavy or very expensive materials. The former makes it much harder to fly while the latter makes it not very commercially viable.
Increased Drag
There's a reason going past the speed of sound was called "breaking the sound barrier." There is a magic number called the Drag Divergence Mach Number (Mach number is the fraction of the speed of sound at which you are traveling). Beyond this number, the drag increases tremendously until you are supersonic, at which point it decreases quite rapidly (but is still higher than subsonic).
Therein lies one of the biggest problems. You need very powerful engines to break the barrier, but then they don't need to be very powerful on the other side of it. So it's inefficient from a weight/cost standpoint because the engines are so over-engineered at cruise conditions (note: this does not imply the engines are inefficient on their own).
Increased Heat
There's no denying that it will get hot. It is storied that the SR-71 would get so hot and the metal would expand so much, that when it was fully fueled on the runway, the fuel would leak out of the gaps in the skin. The plane would have to take off, fly supersonic to heat the skin enough to close the gaps in the metal, then be refueled mid-air again because it used it all up. Then it would go about it's mission.
At the Mach numbers for a commercial aircraft, the heating would not be as extreme. But it would require some careful engineering, which makes it more expensive.
So why can't it just fly higher?
Ignoring international law for a moment, there's several reasons why flying higher just isn't as viable:
- Cabin pressure issues
- Emergency procedures: Let's assume for a moment we could pressurize the cabin. In the event it loses pressure, what do we do? The normal procedure would be to dive down to a safe altitude, that takes considerably longer from 60,000 feet than 30,000 feet.
- Drag is proportional to density, but so is lift. This means to fly higher, an aircraft needs bigger wings. But bigger wings mean more drag, so it gets into a vicious cycle. There is a sweet-spot that can be optimized for an ideal balance, but that means that "just go higher" may not be a good option.
Ceilings and Speeds
This one doesn't have to do entirely with legal issues, but that's part of it. A service ceiling is defined as the maximum altitude at which the aircraft can operate and maintain a specified rate of climb. This is entirely imposed by the aircraft design (laws may require a minimum ceiling, but not a maximum... although they may restrict a plane from flying at the maximum).
Likewise, an absolute ceiling is the altitude at which the aircraft can maintain level flight at maximum thrust. Naturally, as the plane burns fuel and becomes lighter, it needs less lift to stay at the same altitude. But the lift force is based solely on the geometry and speed, so actual lift will exceed what is needed and the plane will climb. As it climbs, the air density drops and so does lift. This means as the plane flies, it's absolute ceiling actually increases.
Now for the speeds... Commercial aircraft fly as close as they can to the Drag Divergence Mach Number because it's the most economic point to fly. The plane goes as fast as it can go without the drag coefficient increasing tremendously. This is usually around Mach 0.8. But they can, and often do, go faster than that.
It's not unusual for an airplane that is delayed taking off to land on time or even early. This happens because they can still go faster than they normally operate (not significantly of course, perhaps Mach 0.83-0.85). It may cost some more fuel because the drag coefficient is likely increasing as it approaches Mach 1, but a delayed plane is more expensive for the airline than the extra fuel used (maybe not in direct dollars, but in PR, reputation, etc.)
The compressor adds energy to the flow, which in turn increases the pressure of the flow. A simple and approximate way of viewing this problem is by using the Bernoulli equation. Consider,
$$P_t = p + \frac{1}{2} \rho \left(u^2 + v^2 + w^2\right) $$
The rotor will add swirl to the flow, which effectively increases the angular momentum of the flow and associated kinetic energy in the tangential velocity component, $\frac{1}{2}\rho v^2$. The stator will remove the imparted swirl from the rotor, but will not add any additional energy to the flow. Thus, the stator can be thought of as converting the kinetic energy of swirl into internal energy of the flow, which is made evident by increase in static pressure of the flow. A conventional velocity and pressure profile across a multistage axial flow compressor looks similar to the following.
More detailed explanations can be obtained in any undergraduate level airbreathing propulsion text. The common analysis encompasses using various reference frames to analyze the true 3-dimensional nature of the flow across a single compressor stage (rotor/stator). These are generally referred to as the throughflow field, cascade field, and secondary flowfield. The analysis also requires velocity diagrams of the velocity components in each subsequent reference frame to which the stage thermodynamic parameters of interest are obtained with the Euler turbine equation.
Also, be careful regarding the shape of a nozzle and its expected role. Depending on the Mach number of the flow entering the nozzle, the shape of the nozzle can have very different effects. A classical results of quasi one-dimensional gas dynamics is the following differential equation termed the area-velocity relation,
$$\frac{dA}{A} = (M^2-1) \frac{du}{u} $$
Major results depending on the Mach number $M$ of the gas go as follows:
1.) $M \rightarrow 0$ (incompressible subsonic limit) suggests that $Au$ = constant. Which is the conventional continuity equation for incompressible flow.
2.) $0 \leq M \lt 1$ (subsonic flow), an increase in velocity (+$du$) is associated with a decrease in area (-$dA$). This is also consistent with the conventional continuity equation for incompressible flows. A decrease in area is accompanied by a increase in velocity, and a increase in area is accompanied by a decrease in velocity.
3.) $M \gt 1$ (supersonic flow), an increase in velocity (+$du$) is associated with a increase in area (+$dA$). This is not consistent with the conventional continuity equation for incompressible flows. For instance, in this case, an increase in area is accompanied by an increase in velocity, and a decrease in area is accompanied by a decrease in velocity.
4.) $M = 1$ (sonic flow), yields $dA/A = 0$, which simply means a minimum or maximum condition in area. The only realizable physical solution is the minimum area condition for which one can choke the flow to the sonic condition at the throat (minimum area) of a converging-diverging nozzle.
Based on the above results, the shape of a nozzle can play varying roles and what sometime may appear as a nozzle actually acts a diffuser. A common schematic demonstrating the above is below.
Best Answer
Intuitive explanation
Imagine you're stuck in a traffic jam on a nice 6 lane highway. As it turns out, up ahead they closed all but one lane so everyone has to merge. As you approach and the number of lanes goes down everyone goes a little faster as the total flow rate of cars must be constant. Now once you pass the narrowest point, instead of everyone slowing back down as they expand back out to 6 lanes they go faster!
This happens because the cars in front have nothing holding them back, so they can accelerate. The flow rate of cars must still be conserved, and this is accomplished by everyone increasing their following distance.
This is almost exactly what happens in the case of a converging diverging nozzle. As the air approaches the throat it speeds up and the pressure goes down and the density goes down. Then after it passes the throat the pressure continues to go down and the air continues to speed up because there is nothing pushing back in front and air molecules naturally want to expand.
Energy conservation
The pressure of air flow is constantly decreasing. This pressure gradient is what accelerates the air.
A converging diverging nozzle is placed after a subsonic combustion chamber to take the high temperature, high pressure gas, and transform it into an atmospheric pressure, high velocity gas that will provide thrust through its high momentum.
As Floris points out, this is a energy conserving transformation: taking energy stored as pressure and heat and turning it into kinetic energy.
Without the high pressure, the gas would not go super sonic and the velocity would just go up and back down as is seen in venturis.