An engineering answer:
Note that it's not just about water vs air. It depends on a lot of things: the density & compressibility of the fluid, and tradeoffs between torque, efficiency, cost, materials, maintenance needs, fouling hazards, and so on. Below is a water turbine blade, the SeaGen, that's not much different from a wind turbine blade, because it's doing a similar job, with similar constraints, but in water rather than air. But bear in mind that this is (AFAIK) the first tidal turbine to achieve commercial grid operation, so later designs could diverge.
Some blades are indeed close to optimised given the materials available at the time, thanks to decades of experience. However, new materials can enable further optimisations to blade design.
There are lots of questions here that I will try to answer, hopefully I'll get to them all...
Creature Comforts
It's hard to "just fly higher" when you consider passenger planes. Supersonic military aircraft like the SR-71 do fly ridiculously high. It's service ceiling is 85,000 feet! But, it has the advantage that it doesn't need to keep anybody but the pilot comfortable. The issue deals with pressurization. As you increase altitude, the aircraft must also be able to withstand a larger pressure differential if the cabin will be kept at a comfortable pressure. Most very high altitude military aircraft do not pressurize the cabin; rather, the pilot wears a pressure suit. Imagine if you had to suit up for a flight to visit relatives!
It's not that we can't build a plane that can withstand the pressure difference, but doing so would require very heavy or very expensive materials. The former makes it much harder to fly while the latter makes it not very commercially viable.
Increased Drag
There's a reason going past the speed of sound was called "breaking the sound barrier." There is a magic number called the Drag Divergence Mach Number (Mach number is the fraction of the speed of sound at which you are traveling). Beyond this number, the drag increases tremendously until you are supersonic, at which point it decreases quite rapidly (but is still higher than subsonic).
Therein lies one of the biggest problems. You need very powerful engines to break the barrier, but then they don't need to be very powerful on the other side of it. So it's inefficient from a weight/cost standpoint because the engines are so over-engineered at cruise conditions (note: this does not imply the engines are inefficient on their own).
Increased Heat
There's no denying that it will get hot. It is storied that the SR-71 would get so hot and the metal would expand so much, that when it was fully fueled on the runway, the fuel would leak out of the gaps in the skin. The plane would have to take off, fly supersonic to heat the skin enough to close the gaps in the metal, then be refueled mid-air again because it used it all up. Then it would go about it's mission.
At the Mach numbers for a commercial aircraft, the heating would not be as extreme. But it would require some careful engineering, which makes it more expensive.
So why can't it just fly higher?
Ignoring international law for a moment, there's several reasons why flying higher just isn't as viable:
- Cabin pressure issues
- Emergency procedures: Let's assume for a moment we could pressurize the cabin. In the event it loses pressure, what do we do? The normal procedure would be to dive down to a safe altitude, that takes considerably longer from 60,000 feet than 30,000 feet.
- Drag is proportional to density, but so is lift. This means to fly higher, an aircraft needs bigger wings. But bigger wings mean more drag, so it gets into a vicious cycle. There is a sweet-spot that can be optimized for an ideal balance, but that means that "just go higher" may not be a good option.
Ceilings and Speeds
This one doesn't have to do entirely with legal issues, but that's part of it. A service ceiling is defined as the maximum altitude at which the aircraft can operate and maintain a specified rate of climb. This is entirely imposed by the aircraft design (laws may require a minimum ceiling, but not a maximum... although they may restrict a plane from flying at the maximum).
Likewise, an absolute ceiling is the altitude at which the aircraft can maintain level flight at maximum thrust. Naturally, as the plane burns fuel and becomes lighter, it needs less lift to stay at the same altitude. But the lift force is based solely on the geometry and speed, so actual lift will exceed what is needed and the plane will climb. As it climbs, the air density drops and so does lift. This means as the plane flies, it's absolute ceiling actually increases.
Now for the speeds... Commercial aircraft fly as close as they can to the Drag Divergence Mach Number because it's the most economic point to fly. The plane goes as fast as it can go without the drag coefficient increasing tremendously. This is usually around Mach 0.8. But they can, and often do, go faster than that.
It's not unusual for an airplane that is delayed taking off to land on time or even early. This happens because they can still go faster than they normally operate (not significantly of course, perhaps Mach 0.83-0.85). It may cost some more fuel because the drag coefficient is likely increasing as it approaches Mach 1, but a delayed plane is more expensive for the airline than the extra fuel used (maybe not in direct dollars, but in PR, reputation, etc.)
Best Answer
If the tips of the propeller blades are moving near the speed of sound a shock wave can form. Supersonic flow has very different character than subsonic. A propeller designed to operate at subsonic speeds will be inefficient at supersonic ones due to shock waves. In general, shock waves cause a loss of efficiency. You might have noticed that subsonic airplanes often have swept wings. This is because flow over the top of the wings accelerates and can reach supersonic speeds even when the aircraft is traveling well below the speed of sound. Sweeping the wings decreases the normal component of the velocity relative to the wing. Likewise, you will sometimes see helicopter blades with the tips swept back. To avoid supersonic flow at the inlet of turbine blades you will observe an increase in area from where air enters the engine to when it reaches the blades. As long as the flow entering the engine is subsonic, the increase in area will slow down the speed of the fluid and avoid supersonic flow over the blades.