The reason being closer to a heat source makes you warmer is the inverse square law. Think of it this way: If you have a $1~\mathrm{m}^2$ piece of material facing the Sun and located at Mercury's orbit, it will be quite hot. What does the shadow of this square look like at Earth's orbit (about $2.5$ times further away than Mercury)? Well, it will be $2.5$ times bigger in both directions, covering about $6~\mathrm{m}^2$. So the same amount of power can be delivered either to $1~\mathrm{m}^2$ on Mercury or to $6~\mathrm{m}^2$ on Earth. Every square meter of Earth gets about $6$ times less Solar power than every square meter on Mercury. The light is not losing energy to the surrounding medium, even if the medium exists.
You were on track...and then missed the mark.
Higher specific heat means that by the end of the day, the plums have stored more thermal energy than the cardboard box."
Correct. You're on track...
Water and metal are good thermal conductors, so they will feel warmer to my hands than the other objects even if they contain the same energy per unit.
But incorrect. You just veered and missed the mark.
You don't feel thermal energy stored in your finger tips; You don't even feel the temperature of the material. You feel the temperature of your fingertips.
This in turn is influenced by the specific heat capacity, thermal conductivity, and actual thermal energy stored in the material.
Of key importance in your scenario is that you are feeling the temperature after the heat source has been removed and things have been given time to cool down. Specific heat capacity does affect how quickly that happens since more energy must be drawn from the material for the same decrease in temperature.
The role thermal conductivity plays is that it determines how quickly your finger tips match the temperature of the material. What this means is that a piece of aluminum (good thermal conductivity) will initially feel hotter than a piece of plastic (bad thermal conductivity) at the same temperature upon initially touching it because it is bringing the temperature of your fingers to match its own faster. But hold your fingers on either long enough, and it will feel the same because your finger has reached the same temperature in either case. This is all assuming the act of touching it does not change the temperature of the object itself since it is transferring heat to or from your fingers after all (see next paragraph).
Specific heat capacity also determines how much an object's temperature changes due to you touching it as heat is transferred to or from your fingers. If an object is small enough it transfers so much of its own thermal energy to your fingers that it drop significantly in temperature while not raising your own hand temperature that much and doesn't burn you. This is the exact same mechanism that enables an object with higher specific heat capacity (for the the same mass) to take longer to cool at dusk since both metal and plum are exposed to the same cooling conditions.
Best Answer
In principle yes, but skin contains many large and complex molecules and thus many chemical reactions can be introduced by absorbing photons. For example, UV light can even damage the DNA (just google for sunburn). Human blood cells absorb blue and green, but reflect red, giving blood its red color (in contrast, plants reflect green thus being green). In general, melanin is the most important substance in human skin interacting with visible light thus producing the color of the skin.
So different substances and different reactions, but the result is that energy is transferred to your skin.
Note, what we call infrared radiation (long wavelengths) does not introduce chemical reactions but just excite rotation and vibration modes, as you mentioned.