I am asking why the cool room warms up faster (loses the cold after you turn off the AC) then the warm room loosing cool down (lose the heat after you turn off the heater).
That's not a universal thing. That depends entirely on the room and the conditions.
But in your description, one big difference is solar heating. When the room cools, (assuming you don't have significant drafts), most of the heat loss will be via conduction through the walls and convection outside. Radiative losses occur as well, but they tend to be much smaller at room temperatures.
On the heating side, the same would be true for the shaded room on a hot day. But if the room isn't shaded, you add in radiative heating. The sun is (in some situations) simply able to deliver a lot more power to your room than your room would dump on mildly chilly day.
Besides the solar heating, it depends significantly on outside temps. Not sure what part of California you are, some places have common summer temps of 100F where you have nearly 30F difference over preferred temp. You mention a heat wave where it might be another 10F above that. It might be much rarer for that location to have a day that much below temp point (say ~44F). So you might be comparing 30F heating days to 18F cooling days.
Besides the outside, you might get heat transfer from other rooms. You said this room is on the second floor. Since heat tends to rise, you might benefit from any warmth in the room below, but not much from any cooling done there.
Finally, it might be a minor point in many places, but nothing running inside your home (except the A/C) is cooling it, and everything is heating it. The refrigerator, the oven, the water heater, the dryer, the big TV, all the occupants, etc are producing heat. For a small space (especially if you're cooking), that can make a difference.
Best Answer
Windows are transparent to visible light, but opaque to infrared. If your curtains are dark, they will absorb the visible and emit infrared. This will cause your house to warm as the light doesn't escape so easily. If they are lighter, they will be more reflective to visible light and will bounce it back through the window. This is a rather gross simplification of the process, but this is the basic idea.