What is the mechanism by which increasing $\rm CO_2$ (or other greenhouse gases) ends up increasing the temperature at (near) the surface of the Earth?
Mostly what I'm looking for is a big-picture explanation of how increasing $\rm CO_2$ affects the Earth's energy transfer balance that goes a step or two beyond Arrhenius's derivation.
I've read Arrhenius's 1896 derivation of the greenhouse effect in section III here. It assumes that
there is non-negligible transmission of the long wavelength radiation from the surface through the full thickness of the atmosphere to space. In the band of $\rm CO_2$ vibrational lines (wavenumbers between about $\rm 600cm^{-1}$ to $\rm 800cm^{-1}$) It is my impression that for most (some? almost all?) of the wavelengths in this band, the atmosphere is optically thick, so the outgoing long wave radiation, e.g. as observed by IRIS on Nimbus 4 had it's "last scattering" somewhere up in the atmosphere, and thus this Arrhenius's "the surface can't radiate into space as efficiently" doesn't apply uniformly across this band. How does this kind of saturation effect modify Arrhenius's description of the greenhouse effect?
If this line of reasoning is correct, then the net outgoing long wave emissions in $\rm CO_2$ band of vibrational lines is some complicated mix of radiation from different altitudes. If my inference is correct, how does this affect the response of the Earth to changes in CO2 concentration?
Maybe there is some sort of statistical-mechanics picture in terms of the photons doing a random walk to escape the atmosphere (for wavelengths where the atmosphere is optically thick), but I don't know how to connect that idea to overall radiative efficiency.
The issue in my understanding that I'm trying to resolve that that Arrhenius's derivation assumes a non-negligible amount of transmission from the surface directly to space. My, admittedly cursory and thus potentially incorrect, understanding of the absorption spectrum of CO2 is that for a range of IR wavelengths the atmosphere (taken as a whole) is effectively opaque. For the portions of the spectrum where there is only some absorption, Arrhenius's argument applies; is the best model to describe the impact of small changes to CO2 concentration to only consider the portions of the IR spectrum that are (partially) transparent and basically ignore the bands that are opaque?
I'm mostly interested in the direct effect of $\rm CO_2$ on an Earth-like planet, so we're dealing with a planet whose blackbody temperature is $\rm \approx 250K$ (in order emit the short wavelength (visible and above) radiation it absorbed from the Sun), but whose surface temperature is more like $\rm 280K$, and has concentrations of $\rm CO_2$ in the $\rm 300ppp-400ppm$ range, but I'm willing to ignore the effects of water vapor (I figure that might overly complicate things), so assuming a dry atmosphere, i.e. just $\rm N_2/O_2$ and $\rm CO_2$, would be fine.
I'm not being cheeky with the "physics grad", assume I know, or can learn, any of the relevant physical or mathematical relationships required to understand the relationship between greenhouse gas concentrations and the heat transfer properties of the Earth.
Best Answer
Executive summary: Carbon dioxide in the atmosphere absorbs some of the energy radiated by the Earth; when this energy is re-emitted, part of that is directed back to Earth. More carbon dioxide $\rightarrow$ more energy returns to Earth. This is the "greenhouse effect".
The full answer is very very complex; I will try a slight simplification.
The sun can be treated as a black body radiator, with the emission spectrum following Planck's Law:
$$H(\lambda, T) = \frac{2hc^2}{\lambda^5}\frac{1}{e^{\frac{hc}{\lambda kT}}-1}$$
The integral of emission over all wavelengths gives us the Stefan-Boltmann law,
$$j^* = \sigma T^4$$
Where $j$ is the radiance, $\sigma$ is the Stefan-Boltzmann constant ($5.67\times10^{-8} ~\rm{W~ m^{-2}~ K^{-4}})$
If we considered the Earth to be itself a black body radiator with no atmosphere (like the moon), then it is receiving radiation from just a small fraction of the space surrounding it (solid angle $\Omega$), but emitting radiation in all directions (solid angle $4\pi$). Because of this, the equilibrium temperature for a black sphere at 1 a.u. from the sun can be calculated from Stefan-Boltzmann:
$$4\pi \sigma T_e^4 = \Omega \sigma T_s^4\\ T_e = T_s \sqrt[4]{\frac{\Omega}{4\pi}}$$
Now the solid angle of the sun as seen from Earth is computed from the radius of the sun and the radius of the Earth's orbit:
$$\Omega = \frac{\pi R_{sun}^2}{R_o^2}$$
With $R_s\approx 7\times 10^8 ~\rm{m}$ and $R_o\approx 1.5\times 10^{11}~\rm{m}$ we find $\Omega \approx 5.4\times 10^{-5}$; given the sun's surface temperature of 5777 K, we get the temperature of the "naked" earth as
$$T_e = 278~\rm{K}$$
[updated calculation... removed a stray $4\pi$ that had snuck in to my earlier expression. Thanks David Hammen!]
Note that this assumes that the Earth is spinning sufficiently fast that the temperature is the same everywhere on the surface - that is, the sun is heating all parts of the Earth evenly. That is not true of course - the poles consistently get less than their "fair share" and the equator more. Taking that into account, you would expect a lower average temperature, as the hotter equator would emit disproportionately more energy (the correct value for the "naked earth black body" is 254.6 K as David Hammen pointed out in a comment); but the (relatively) rapid rate of rotation, plus presence of a lot of water and the atmosphere does prevent some of the extreme temperatures that you see on the moon (where the difference between "day" and "night" can be as high as 276 K...)
Now we need to look at the role of the atmosphere, and how it modifies the above. Clearly, we are alive on Earth, and temperatures are much higher than would be calculated absent an atmosphere. This means the "greenhouse effect" is a good thing. How does it work?
Now for the "greenhouse effect". I already mentioned that clouds stopped some of the Sun's light from reaching the Earth's surface; similarly, the radiation from Earth will in part be absorbed/re-emitted by the atmosphere. The critical thing here is absorption followed by re-emission (when there is equilibrium, the same amount of energy that is absorbed must be re-emitted, although not necessarily at the same wavelength).
When there is re-emission, some of the photons "return" to Earth. This has the effect of making the fraction of "cold sky" that the Earth sees smaller, so the expression for the temperature (which had $\sqrt[4]{\frac{\Omega}{4\pi}}$ in it) will be modified - we no longer "see" $4\pi$ of the atmosphere.
The second effect is absorption. The absorption spectrum of $\rm{CO_2}$ can be found for example at Clive Best's blog
As you can see, much of the energy emitted by Earth is absorbed by the atmosphere: $\rm{CO_2}$ is not the only culprit, but it does have an absorption peak that is quite close to the peak emission of Earth's surface, so it plays a role. Increase the $\rm{CO_2}$ and you increase the amount of energy that is captured by the atmosphere. Now when that energy is re-emitted, roughly half of it will be emittend towards the Earth, and the other half will be emitted to space.
As energy is re-emitted back to Earth, the effective mean temperature that the surface has to reach before there is equilibrium (given a constant influx of energy from the Sun) goes up.
There are many complicating factors. Hotter surface may mean more clouds and thus more reflected sunlight; on the other hand, increased water vapor also implies increased absorption in the IR.
But the basic idea that absorption of IR by the atmosphere will lead to an increased equilibrium temperature of the surface should be pretty clear.
Update
The question "If the atmosphere is already so opaque to IR radiation, why does it matter if we add more CO2?" deserves more thought. There are three things I can think of.
Spectral broadening
First - there is the issue of spectral broadening. According to this lecture and references therein, there is significant pressure broadening of the absorption lines in $\rm{CO_2}$. Pressure broadening is the result of frequent collisions between molecules - if the time between collisions is short compared to the lifetime of the decay (which sets a lower limit on the peak width), then the absorption peak becomes broader. The link gives an example of this for $\rm{CO_2}$ at 1000 mb (sea level) and 100 mb (about 10 km above sea level):
This tells me that as the concentration of $\rm{CO_2}$ in the atmopshere increases, there will be more of it in the lower (high pressure) layers, where it effectively has no "windows". At lower pressures, the gaps between the absorption peaks would let more of the energy escape without interaction. This will be more important in the upper atmosphere - not so much near Earth's surface where pressure broadening is significant.
Near IR absorption bands
In the analysis above, I was focusing on the radiation of Earth, and its interaction with $\rm{CO_2}$ absorption bands around 15 µm - what is usually called the "greenhouse effect". However, there are also absorption bands in the near-IR, at 1.4, 1.9, 2.0 and 2.1 µm (see Carbon Dioxide Absorption in the Near Infrared. These bands will absorb energy of the sun "on the way down", and result in atmospheric heating. Increase the concentration of carbon dioxide, and you effective make the earth a little better at capturing the sun's energy. In the higher layers of the atmosphere (above the clouds) this is particularly important because this is energy absorbed before clouds get a chance to reflect it back into space. Since these bands have lower absorption (but the incident flux of sunlight is so much higher), they play a role in atmospheric modeling (as described more fully in the paper linked above).
More absorption from "side bands"
This is really well explained in the answer by @jkej but worth reiterating: beside the spectral broadening that I described above, given the shape of a spectral peak, the lower absorptivity as you move away from the center frequency becomes more significant as the total number of molecules increases. This means that the part of the spectrum that was only 10% absorbed will become 20% absorbed when the concentration doubles. As the linked answer explains, this only leads to a "square root of concentration" effect for a single line in the spectrum, and an even smaller amount when spectral lines overlap - but it should not be ignored.
I think there may also be an argument that can be made regarding treating the atmosphere as a multi-layered insulator, with each layer at its own temperature (with lapse rate mostly controlled mostly by convection and gravity); as carbon dioxide concentration increases, this will change the effective emissivity of different layers of the atmosphere, and this might expose the surface of the earth to different amounts of heat flux depending on the concentration. But this is something I will have to give some more thought to... and maybe run some simulations for.
Finally, in a nod to "the other side", here is a link to a website that attempts to argue that carbon dioxide (let alone man-made carbon dioxide) cannot possibly explain global warming - and that global warming in fact does not exist at all. Writing a full refutation of the arguments in that site is beyond the scope of this answer... but it might make a good exercise for another day.