I'll just go ahead and write this down even though it's already been covered in the other thread.. but I didn't post there so :)
First, if you think of light as a (scalar) wave (which is really a semi-classical way of thinking but might be enough to answer your question) you can invoke the Huygen-Fresnel principle which in this case boils down to considering every point on the reflecting surface as an origin of a re-emitted spherical wave with a starting phase directly related to the phase the point got from the incident wavefront.
The superposition of these wavefronts, after you let them destructively interfere with each other, will amount to a new combined wavefront which propagates according to Snells law (angle of incidence = angle of reflectance). See this image for the corresponding illustration of refraction (which is very similar, I couldn't quickly find a good image of reflection):
Now, light really doesn't behave "sometimes as a particle, sometimes as a wave". It is always detected as quanta (particles) but the probability amplitudes (phases) propagate in a wavelike fashion. One way of expressing the propagation is to say that a photon is sort of split and takes every possible path between A and B (or, in the case of a reflector, from A to any point on the reflector and then from there to point B by any means). Every path gets a phase contribution, and all the indistinguishable paths are summed. Most paths are simply cancelling each other but some constructively interfere, creating a large contribution (in case you don't know QM, the probability amplitude squared is the probability of the described event so a large contribution means this result will most probably occur). There is a VERY good image and description of this process in Feynman - QED The Strange Theory of Light and Matter (as I wrote in the comment above).
In the case of the reflector, the large contribution occurs at the classical angle of reflectance (Snell's law again). Notice the similarity between this formulation (called the path integral approach) and the semi-classical principle outlined above; this is not a coincidence of course.
Also to briefly digress on your implied question on the per-atom non-zero "reflectance" time - saying that an electron-orbit absorbs the photon energy for a while the re-emits it a non-zero time later is of course also a slight simplification. In reality the electron interacts with the a photon, changes its momentum a bit, the re-emits (interacts) with the new photon and changes its momentum again. This scattering process occurs at all allowed momenta and intermediate times, which are then all superpositioned like above and hence I'm not sure it's meaningful to talk about any appreciable time of reflectance. Notice that this scattering is in practice very different from scattering which can excite the electron to another orbit.
The easy answer is to say that Brewster's law only applies to reflection from the interface with a transparent medium, and a mirror isn't transparent. Indeed for an ideal perfect mirror, all light of both polarizations is reflected perfectly, so there is nothing to say.
For an actual real-world mirror, the metal mirror surface will have a finite skin depth, and can be considered a dielectric medium with a very large, complex index of refraction. This does lead to a small polarization dependence of the reflection coefficient for near grazing incidence angles. The analogue of Brewster's angle occurs at an angle given by 2*pi*(skin depth) / wavelength above grazing, where the parallel polarized reflection coefficient will reach a minimum ( but not zero, still close to 1). For more details, see for example Landau and Lifshitz, volume 8, section 87.
Best Answer
The reflection could be viewed as a two step process. The incident wave causes the electrons in the silver to vibrate like in an antenna. Though by vibrating they also emit the same light. So it's the electrons at the surface of the silver that reflect the incoming wave. As you mentioned the wave is part electric and part magnetic, but these cannot be taken apart since they are each others cause and effect: without one the other wouldn't be there either, and therefore it must reflect both parts.
That silver (and all metals) don't distort is due to the fact that they are also very good conductors. This prevents the electromagnetic waves from entering the object. The boundary conditions which must hold (from being an conductor) result in the perfect reflection and that the resulting angle is equal to the incident angle.
Similar boundary conditions are there for non-conducting materials like plastic and glass. These similar conditions result in reflection of glass and the shine/reflection on other smooth surfaces (though there can be other causes too). Also Snell's law would follow from these boundary conditions.
In contrast to conducting materials it is possible for electromagnetic waves to enter non-conducting objects. As a consequence part of the incoming wave is transmitted into the material. The propagation or dampening of the wave through the material is largely dependent on the properties of the material. Some materials like glass hardly dampen the wave and you can see through them, while others like most plastics dampen them and thus are opaque.