The color of a surface doesn't reliably indicate the emissivity at non-visible wavelengths. The color in the visible spectrum is more of a side effect than anything. Most thermal radiation around body temperature or room temperature happens in the infrared region, not the visible, and that's not reliably indicated by visible color:
(Source: JPL)
The visibly transparent glasses are opaque to the body's infrared emissions, while the visibly opaque trashbag is transparent to infrared. So one property has no relation to the other.
The emissions of the sun occur mostly in the visible region, which is why white clothing reflects solar energy and stays cool while black clothing absorbs solar energy and gets hot:
(Source: User:Dragons flight)
But your body's thermal radiation is in the infrared, so this rule doesn't apply to the inside of clothing (unless your body is hot enough to radiate visible light, but then you have bigger problems).
Your basic idea would work, though, if you found a material that reflects visible light while transmitting infrared light (but that material would probably have the same properties on the inside and outside, and thus be visibly white on the inside, too).
For example, white paint is quoted as having an absorptivity of 0.16, while having an emissivity of 0.93. This is because the absorptivity is averaged with weighting for the solar spectrum, while the emissivity is weighted for the emission of the paint itself at normal ambient temperatures. ... The white paint will serve as a very good insulator against solar radiation, because it is very reflective of the solar radiation, and although it therefore emits poorly in the solar band, its temperature will be around room temperature, and it will emit whatever radiation it has absorbed in the infrared, where its emission coefficient is high. − Kirchhoff's law of thermal radiation
NASA uses such materials, which they call "selective surfaces", and are used to cool the Hubble telescope:
These surfaces can be designed to reflect solar radiation, while maximizing infrared emittance, yielding a cooling effect even in sunlight. On earth cooling to -50 °C below ambient has been achieved, but in space, outside of the atmosphere, theory using ideal materials has predicted a maximum cooling to 40 K!
Wikipedia's article on selective surfaces describes the opposite effect: Transmitting sunlight and blocking infrared from escaping, to capture the sun's energy.
This article has some relevant results based on a study of bird plumage (it also happens to be cited in the abstract of the Nature paper mentioned in one of the other answers), and is summarized in simpler terms here.
I'll attempt to summarize the summary.
Black and fluffy/loose fitting clothing is best if it is hot out and there is any ($>3 \mathrm{m}/\mathrm{s}$) wind. The black clothing absorbs both solar radiation and radiation from the body. The air in the immediate vicinity is heated, then efficiently transported away by the wind. This is slightly better than white fluffy/loose fitting clothing, which reflects more sunlight and radiation from the body. The emission from the body is reflected, so it cannot heat the air near the clothing as efficiently and have a chance to be transported away.
Tight black clothing is a terrible idea if trying to stay cool, regardless of windspeed.
If there is no wind ($<3 \mathrm{m}/\mathrm{s}$), white clothing is better since the most important thing in these conditions is to reflect as much incoming sunlight as possible.
I also have another possibility to think about. My recollection regarding loose fitting black robes in the desert is that - given a garment that is open at the bottom (robe) and top (not too tight fitting) - heating the air inside is actually advantageous to keeping cool since this drives a convection flow upward through the garment. This airflow makes cooling via sweating efficient, enough that the person wearing the garment doesn't feel as hot. Unfortunately I can't find any experimental results to validate this picture, but it seems more or less in line with the results above, at least in as much as airflow seems to be key to answering the question.
Best Answer
The term "light" is a little ambiguous, because for some it means visible light, and for others it means any form of electromagnetic radiation. But I agree with you, that black and white should be more about visible light than infrared radiation. If we are right, then it's better to wear white, because white will reflect the incident visible sunlight, while not doing anything different with the infrared radiation from our bodies. I suspect the conventional wisdom may have got this one right, and the "correction" to it might be in err. Perhaps it needs to be an episode on "Mythbusters."