I'm having difficulty understanding the full implications of the Lyman alpha forest and its use in cosmology.
My understanding is this: we detect features in the Intergalactic Medium (IGM) by very bright and very far away quasars via IGM absorption lines in the spectra of these quasars. If the IGM was made up primarily of neutral hydrogen HI, we would see the strongest absorption line as the 1s to 2p transition, but naturally redshifted. So, viewing this absorption trough in the spectrum of quasars gives us an estimate of the amount of neutral hydrogen HI in our line of sight.
However, there is not a continuous absorption trough but rather a series of absorption lines, i.e. the Ly-$\alpha$ forest. Therefore, we conclude there is NOT a uniform distribution of neutral hydrogen HI along the line of sight, but rather a series of clouds of HI and that there is very little HI gas outside these clouds in the IGM. This lack of HI is due to "reionization" of HI from formation of the first galaxies, stars, and quasars.
How does one derive the redshift when reionization happened, around $z\approx6$? And I still don't understand the implications to cosmology. What's the Gunn-Peterson trough? Papers appreciated.
Best Answer
Lyman Alpha absorption systems
The Lyman-$\alpha$ Forest and Gunn-Peterson troughs are two extremes on the scale of absorption features that are left by neutral hydrogen in intergalactic space.
When ultraviolet light from a background source, typically a Quasar or a young, strongly star forming galaxy, travels through intergalactic space, it is continuously redshifted on the way towards our detectors. When encountering systems of HI on its way, as is shown in this this video.
What is shown in these videos are only Ly$\alpha$ forest systems, that is, hydrogen clouds with a column density of less than $N \approx 10^{16}$. Denser systems leave deeper, broader absorption profiles in the UV continuum of the source, so called Lyman Limit ($10^{16} < N < 10^{21}$) or Damped Lyman Alpha ($N > 10^{21}$) systems; but the principle is the same: the wavelength of each absorption feature reveals the redshift and thus distance from us (and the source) at which it resides. An example of a $z \approx 3$ Quasar spectrum with a strong Ly$\alpha$ Forest and two DLA systems at redshifts 2.4 and 2.5 is shown here:
Already from this, a couple of cosmological applications are available:
The wavelength-distance mapping allows of to map the density of systems of different mass at different redshifts along a certain line of sight. These HI clouds are distributed by falling into the potential wells of Dark Matter, they trace the mass distribution of the Universe, not just the distribution of luminous matter as emitting galaxies or Quasars do.
The densest of these systems, the DLA's trace early galaxies. While they are much rarer than the Ly$\alpha$ Forest systems and Lyman Limit systems, they are much denser and contain the large majority of the neutral gas in the Universe - so they MUST be the reservoirs from which galaxy formation material is drawn. However, DLAs are biased towards systems of larger extent, which is not the same as the more luminous systems. Therefore, DLAs provide a valuable complementary way of mapping the galaxy formation of history, helping us overcome the inherent selection biases that comes from observing galaxies in emission. Since galaxy formation is also coupled to the Dark Matter distribution in the Universe and its evolution with redshift, this can also help us constrain Dark Matter models in much the same way that studies of Lyman Alpha Emitters, Lyman Break Galaxies, Luminous InfraRed Galaxies (LIRGs) can do, with each their different bias.
Gunn-Peterson Troughs and the Epoch of Reionization
As the OP states, the Universe was largely neutral From around 300.000 years after the Big Bang - the Cosmic Microwave Background is emitted at this time, as this was the first time the Universe got transparent to the wavelength of the CMB at the time of emission, and these photons could travel freely. However, the Universe was not transparent to radiation with wavelengths shorter than the ionization wavelength of Hydrogen (912 Å), or to the strong transitions in Neutral Hydrogen, of which Ly$\alpha$ is by far the strongest. More importantly, light emitted from a hypothetical source in this neutral Universe would be redshifted as it travelled through it, meaning that all light bluewards of Ly$\alpha$ would be continually redshifted into absorption, leaving only flux on the red side of Ly$\alpha$ line center. However, at some redshift the Universe is completely reionized, after which photons bluer than Ly$\alpha$ at this redshift are unaffected and can travel freely. The result is a deep trough of almost zero flux ranging from Lyman$\alpha$ at the redshift of the emitter to Lyman$\alpha$ at the redshift where reionization happens.
However, as the OP states, reionization is not a neither smooth, homogeneous nor instantaneous process, so the above view is strongly idealized. Reionization started already with the formation of the first stars, at least as early as at redshift $z=11$, probably higher. It started out in small bubbles that grew, started overlapping and in time left only little "islands" of neutral Hydrogen behind, like is illustrated here (taken from a great review paper by Mark Dijkstra):
In the above figure, reionization seems complete already around $z \approx 8$, but there is still enough neutral hydrogen floating around to make it opaque to Ly$\alpha$ - as small a neutral fraction as $\sim 10^{-3}$ is enough, so Ly$\alpha$ marks only the very end of the process of reionization, around $z \sim 6$.
The way this shows in our spectra is, as we go towards higher redshifts, that the Ly$\alpha$ forest systems become denser and denser in our spectrum until they start overlapping and only leaving a small fraction of the intrinsic flux bluewards of Ly$\alpha$. When exactly we move from a very dense Ly$\alpha$ fores to a bona fide Gunn-Peterson Trough is a bit wishy-washy when looking in the specrum. Below is a figure from the first paper reporting obervation of a G-P trough; only the lowest panel is deemed a G-P trough by the authors.
Interestingly, G-P troughs observed in Quasar and Galaxy spectra report slightly different values of the redshift at the end of reionization. This is because Quasars tend to cluster in the regions of highest density, which reionize first. These quasars can reside in huge bubbles that are reionized ealier than the Universe as a whole, and their Ly$\alpha$ line can be redshifted into "safety" before encountering any neutral hydrogen. Young Ly$\alpha$ emitting galaxies are a sligtly better tracer of the general reionization state of the Universe, as is described in more detail in Mark Dijkstra's review paper.
A proper determination of this redshift, of course, also needs to be averaged over a large number of sight lines.
Summary