You are seeing only one color from each droplet. Even though each droplet reflects the whole spectrum, only one color gets to your eye. The rest of the light from a single droplet is sent somewhere else than your eye (maybe into the eye of someone standing near you). So if you see a thick blue band in a rainbow, the blue band is formed by many reflections from many droplets. You cannot see many thin rainbows (each one from a single droplet), because each droplet contributes to your complete rainbow image as one "pixel" only, in a sense.
Each individual droplet reflects a whole cone of light - a complete circular rainbow (the image below has only 2 colors for clarity, but you get the idea - see this video):
All droplets in a rain reflect a huge amount of these cones (circular rainbows), but you can see only a small fraction of every cone from each droplet. All these fractions sum into the rainbow you see. So - how is the vertical portion of a rainbow formed? Simply - it is formed by the rays from the vertical portions of the light cones reflected by the droplets that are located in the direction where you see the vertical portion of the rainbow.
Again - the rays that you do not see are sent somewhere else. You can see them if you move around, but then you cannot see the rays that you were seeing in your previous location.
That means that a single cone from a single droplet can contribute to different parts of different rainbows, seen by different people: it can contribute to the upper horizontal part of a rainbow seen by you (that would be a color ray from the lower section of that individual cone), but at the same time it can contribute to the vertical portion of a rainbow seen by someone else who is higher and to your left (that would be a color ray from the left vertical portion of that individual cone).
There are two kinds of circular polarizers: right (clockwise) and left (counterclockwise). The first will transmit right circularly polarized light and absorb left circularly polarized light; the second does the opposite.
Turn the glasses over so the light goes the other way, and you will see that the effect is different.
A quarter wave plate, as @Jasper pointed out, is always wavelength (and light propagation angle) dependent. Most polarizers work less well with shorter wavelengths.
If light is randomly polarized, then passes through a linear polarizer, it becomes linearly polarized. Then when the light subsequently passes through the quarter-wave plate, the linear polarization converts to circular polarization. A quarter wave plate has two axes: a "slow" axis and a "fast" axis. Light propagates a little bit slower if it is polarized in the "slow" direction. The thickness of the quarter wave plate is precisely enough to slow light polarized in the "slow" direction, by a quarter of a wave (relative to light polarized in the "fast" direction. If you draw the waves, you'll see that the QW plate does nothing to light that is polarized precisely along either axis; but if light entering the QW plate is polarized at 45 degrees it comes out circularly polarized in, say, the clockwise direction. Rotate the QW plate by 90 degrees and the light will come out circularly polarized in the counterclockwise direction instead. If light entering So, the orientation of the QW plate relative to the linear polarizer is crucial, but when light enters from the linear polarizer side everything works right because the linear polarizer absorbs the portion of light that's polarized "wrong".
On the other hand, when randomly polarized light enters from the QW plate side, essentially nothing happens to it because it is still randomly polarized when it exits the QW plate. When it subsequently enters the linear polarizer, the light simply comes out polarized in the direction determined by the orientation of the linear polarizer.
Now consider what happens if circularly polarized light passes through from either side. If it enters from the QW plate side, *and if the handedness (clockwise or counterclockwise) is right, the QW plate converts it to linearly polarized light that passes almost 100% through the linear polarizer. If the handedness is wrong, the QW plate converts it to light that is linearly polarized light but in the wrong direction so it gets nearly 100% absorbed by the polarizer. Rotating the polarizer makes no difference to the amount of light that passes through, but it affects the polarization angle of the exiting light.
However, if circularly polarized light enters from the linear polarizer side, the polarizer absorbs the component that's not aligned with the polarizer, then converts the remainder to circularly polarized light with a certain handedness. Rotating the circular polarizer in this case makes no difference.
The third case to consider is that of linearly polarized light entering the filter from the QW side or linear polarizer side. If linearly polarized light enters from the QW side, the QW plate converts it to elliptically polarized light: circularly polarized if the linearly polarized incident light is at 45 degrees to the axes of the QW plate. The linear polarizer then transmits the portion of the elliptically polarized light that has the proper linear polarization. If linearly polarized light passes first through the linear polarizer, then through the QW plate, the linear polarizer passes only the fraction of the light having the proper linear polarization (note: linear polarization can be treated as a vector, so light polarized at 45 degrees from the vertical is equivalent to two vector components polarized vertically and horizontally, each with amplitude $1/sqrt(2)$). The QW plate then converts the passed light into right- or left-circularly polarized light.
There is a really good Harvard lecture here about polarization. Lots of good graphics in the Wikipedia article on circular polarization.
You will get color effects whenever different wavelengths are affected differently. In your experiment, you found that rotating the 3D glasses 90 degrees to the screen's vertical axis changed the color of light passing through the glasses. That means that the light emitted by the screen is polarized differently for different wavelength ranges. You don't know without further experimentation, in what way the polarization differs by wavelength. However, if you turn your 3D glasses over, they will act effectively as linear polarizers. You can explore the linear polarization of different wavelength ranges by putting colored cellophane between the polarizer and your eye as you rotate the polarizer while looking at the screen.
To see some really beautiful effects, look at some wrinkled cellophane tape between two crossed linear filters.
Best Answer
What you are seeing is stress in the window resulting in birefringence: the speed of propagation of polarized light depends on the direction of polarization.
In the setup you have, the light in the sky is partially polarized because that's how Rayleigh scattering works; this partially polarized light is transmitted through the window where it rotates (because of the birefringence) depending on the stress. The polarizer on your camera acts as the analyzer: some of the polarized light will be more at right angles while other light is more parallel to the axis of the second filter.
Now birefringence is a function of wavelength: so different colors will be rotated by different amounts, and will be more or less attenuated. And this is what gives rise to the colors.
Here is an example of an image of a plastic box with in built stresses viewed through crossed polarizers source:
UPDATE - why Rayleigh scattering leads to polarized light:
In this website we read:
In Rayleigh scattering, the electrons around an atom are a driven simple harmonic oscillator: classically, you can think of it as a negative cloud that can move with respect to a positive center, and if you could displace it, it would vibrate around its equilibrium position with some frequency $\omega_0$. Now when you excite this cloud with a transverse electrical signal (EM wave like light) it will emit light mostly at right angles to the axis of excitation - in fact there's a $\left(\frac{\omega}{\omega_0}\right)^4\sin^2\theta$ term in the intensity distribution. This both tells us that the intensity of the scattered light drops quickly for longer wavelengths (the key is blue) and also that when the sun is to your right, the polarization of the sky will be in the up/down direction (perpendicular to the line from the sun to the point). This is explained in more detail at this website on polarization which is also the source of this animation that shows the direction of polarization that you expect: