The term "light" is a little ambiguous, because for some it means visible light, and for others it means any form of electromagnetic radiation. But I agree with you, that black and white should be more about visible light than infrared radiation. If we are right, then it's better to wear white, because white will reflect the incident visible sunlight, while not doing anything different with the infrared radiation from our bodies. I suspect the conventional wisdom may have got this one right, and the "correction" to it might be in err. Perhaps it needs to be an episode on "Mythbusters."
IR is an extremely broad wavelength range (from near infrared, 800 nm to far infrared up to 1000000 nm). What you deal with when you experiment with a TV remote and a phone camera is near infrared (800-1000 nm). The material properties in that range are still pretty comparable to the optical range (although we don't see "infrared colors"). With a near-infrared filtering camera you will be surprised to see that many colored objects appear white, though. This is especially striking for many black colored objects. The reason is probably simply that the designed objects are not "optimized" for the NIR range because nobody would see those "colors" anyway, and wouldn't pay for it.
However, this has nothing to do with far infrared heat radiation (or is rather only a tiny part of it) which starts at about 3000 nm up to 50000 nm. Absorption spectra in that range are totally unrelated to the near infrared. The latter is still dominated by electronic transitions (similar to the colors we see) inside molecules, while the former is dominated by the vibrations (or phonons, quantum-mechanically) between the molecules.
That is the reason why you cannot tell anything about the thermal radiation behavior (especially reflectivity, or "FIR color") of clothes by referring to their various visible colors. It's totally different physics, so to say. The only notable exception are metals: they reflect well in the optical and NIR range, and so do they in the FIR range. This is the reason why it is very difficult to determine the temperature of metal objects with a thermal camera (you might just see the reflections of nearby objects, e.g. yourself, and hence, their temperatures instead of the true temperature of the metal part).
What does have an indirect thermal effect, though, is the absorption of visible light and NIR and the subsequent conversion into heat radiation (much larger wavelength) due the temperature of the object being raised. Black clothes absorb the light of the sun very effectively and raise your temperature a lot while you are in the sun, as opposed to white clothes, which reflect most of the sunlight, and hence, allow less to be converted into heat.
I would claim that what you have read in your book is a very popular science myth, set up to explain the confusing fact that people in the middle-east or africa wear black clothes, where this is actually probably just an accidental cultural mannerism, rather than a kind of "intuitive science by wise ancient grey-bearded sufi shamans".
Best Answer
The color of an object does give some idea as to how much it absorbs or reflects EM energy, but only radiation in the visible light spectrum.
Where visible light is an important part of the heat transfer (such as in direct sunlight), then white objects would be expected to absorb less energy (and heat up less) than dark objects.
But since underwear is (normally) worn beneath mostly opaque clothing, the heat transfer doesn't involve visible light photons. The perceived warmth would have more to do with how it conducts heat directly and how well it absorbs/radiates in the infrared spectrum. Neither of these correlate with the color of the material.
So no, I would not expect white underwear to be more or less insulating than black.