Some Background
I became fascinated with how the overtone series, or harmonics, relates to how brass instruments function. Most trumpet players (or brass, really) should notice that as they play higher notes, they become closer together on the staff. What fewer might notice is the pattern, where each octave that one goes up, there becomes twice as many notes (played without valves), and that each newly available note comes in between the notes of the previous octave.
This led me to notice one last thing: trumpets can't play the fundamental frequency for their overtone series. The lowest note one can clearly produce (without using the valves) is a middle C, but the next note up is a G, clearly not the 2nd harmonic. This means that the lowest clear note for a trumpet is the 2nd harmonic, not the fundamental.
However, I have also played the flugelhorn, which has the same tube length as a trumpet, and the same overtone series (the 2nd and 3rd harmonics are still the same frequency C and G as on a trumpet). On a flugelhorn, I found that you can also comfortably drop an octave below middle C, which is the fundamental frequency for the instrument.
The question:
Why can a flugelhorn clearly and easily drop to the fundamental, when a trumpet can not? The main functional difference between the two instruments is that a trumpet has a cylindrical bore while a flugelhorn has a conical bore. Why does this difference in bore have such an impact on the range that each instrument can play? I know that the cylindrical bore has a "warmer sound", but I don't particularly care about that for the purposes of this question.
I did notice a section of this Wikipedia article mentions "whole tube vs. half tube" in relation to the ability of some brass instruments' ability to hit their fundamental, but I am interested in a more in-depth explanation of why that is important.
Visuals of the instruments in question
You can clearly see the difference in tubing between the two instruments.The trumpet's diameter only widens along its final approach to the bell, while a flugelhorn's diameter widens along the entire length of its tubing. Otherwise, they play nearly identically (outside of the thing that sparked this entire question).
Best Answer
In a simplified model, the tube is effectively closed by the player's lips at the mouthpiece end. For the theoretical case of a straight tube with completely cylindrical or conical bore, and no bell, the harmonics of the conical bore have frequency ratios 1:2:3:4.... but the cylindrical bore only has the odd harmonics 1:3:5:7....
Those facts are most obvious in woodwind reed instruments. The clarinet has a cylindrical bore and the first harmonic is at the 12th (3 times the fundamental frequency) but almost all other instruments (e.g. the saxophone) have conical bores and the first harmonic is at the octave (2 times the fundamental).
It is also apparent in organ reed pipes, where the pipes designed to imitate trumpet tone are literally conical along their entire length, with no bell.
In "cylindrical bore" brass instruments, the large bell reduces the frequency of each harmonic compared with the simple theory, so the actual frequencies change from 1:3:5:7:... to something close to (a number less than 1):2:4:6:... The actual shape of the "cylindrical" tube is carefully designed to make the higher ratios as close as possible to 4:6:8:... and so far as the player is concerned, these are equivalent to the 2:3:4:... of a conical tube of twice the length. Hence the description of "half tube" instruments which are (approximately) half the length of a "whole tube" instrument that from the players point of view sounds the same harmonics.
However the fundamental frequency can not be accurately "modified" in the same way as the higher harmonics. It is always physically playable, but it may be so out of tune, and its frequency very unstable and depending on the exact tension in the player's lips when blowing, that it is musically useless and hence described as "unplayable" in practice. The embouchure for playing the fundamental may be quite different from playing the higher harmonics, so there is no practical use for learning how to play a note that is completely out of tune in any case!
It's worth noting that the physicist Helmholz who founded the modern study of acoustics in the 19th century got his theory of brass instruments completely wrong, because he didn't realize that the player's lips effectively formed a closed end to the tube and not an open end. And his reputation as a scientist was so great that it was half a century or more before anyone seriously questioned his incorrect theory! Of course musical instrument makers had been constructing practical instruments for centuries before that, and musicians had been playing them, with no theory at all except the empirical observation of the frequencies of the different harmonics.