As previous answers have stated, the wavelength (or frequency) and intensity of the beam are important, as well as the type and amount of impurities in the air. The beam must be of a wavelength that is visible to humans, and fog or dust scatters the light very strongly so that you can see it. However, even in pure, clean air, you will be able to see a laser beam under certain conditions.
This is because light can scatter from air molecules themselves via Rayleigh scattering. Rayleigh scattering has a strong inverse dependance on wavelength, specifically $\lambda^{-4}$, so it will be easier to see with a green, and especially a blue, laser1. It also has a scattering angle dependance that goes like $1+\cos^2 \theta$, so it may be easier to see if your viewing angle is very close to the beam2.
With a 5mW green laser pointer, Rayleigh scattering is pretty easy to see. I imagine it would be even easier with blue/violet, but I'm not sure, since human eyes are most sensitive at green, so that may tip the balance. A more intense beam, like those used at night clubs or laser light shows, would be very easy to see if the beam were held still, but in those situations the beams are moving around rapidly to produce the light show, so Rayleigh scattering alone wouldn't really let you see much. In situations like night clubs, the scattering from fog produced by fog machines is much more important.
You are correct that, in space, because there is no atmosphere and nothing to scatter off of, you wouldn't see any sort of laser beam.
1: This is also why the sky is blue, incidentally.
2: DO NOT EVER TRY TO TEST THIS WITH A BEAM POINTED TOWARDS YOU If you want to try this out, take a laser pointer and hold it near your head (eg. against your temple) and point it away from you, in the dark.
Best Answer
Gases are very inefficient emitters of black body radiation so they do not have the colour expected for the temperatures of their flames.
The main mechanism for the emission of black body radiation are transient fluctuations in electron density due to thermal vibrations. However this mechanism is specific to solid or liquids (with a few more exotic special cases e.g. plasmas). Gases are simply not dense enough to produce the type of oscillations in electron density needed to produce black body radiation.
As you say in your question, If solid particles such as soot are present then the flame will heat these and they can produce black body radiation to give the characteristic yellow flames. However generation of soot requires specific conditions i.e. insufficient oxygen for complete combustion.
The reason flames such as from burning hydrogen produce any light is that the temperature rise causes an increase in the average velocity of the gas molecules. The velocity distribution is related to the gas temperature by the Maxwell-Boltzmann equation, and while the vast majority of gas molecules have energies around $kT$ a small proportion of them have enough energy for their collisions to cause electronic transitions. The excited atoms/molecules then decay by emission of a photon, and it's the light produced from these decays that we see as the colour of the flame.
You say that hydrogen flames are visible, but in fact a flame of hydrogen burning in air emtis almost no light in the visible region. There is an emission band in the near ultraviolet at around 300nm and strong emission in the infrared due to vibrational excitations of the water produced, but almost no visible light. The orange flame in the video you link of the bursting ballons is likely to be due to soot from burning rubber.