I believe your puzzlement comes from confusing two frameworks: the quantum mechanical (photons) and the classical mechanics one, waves.
When one is calculating in terms of classical electromagnetic waves there are classical considerations : refraction, absorption, reflection with their corresponding constants .
When one is zooming in the microcosm and talking of photons, a wave is composed of zillions of photons which go through, each at the velocity of light.
The bulk of the target material is in effect the electric and magnetic fields holding the atoms together to form it: the nuclei are tiny targets and the electrons are small zooming targets. The probability of a single photon to scatter on a nucleus or an electron is miniscule. It interacts/scatters out of its optical ray path with the electric/magnetic fields that are holding the glass or crystal together. The scattering angles are very small in transparent materials thus preserving the optical path, enormous in opaque ones . It is those fields that one has to worry about, not the individual atoms and their excitations.
The photons scatter mostly elastically with the fields holding the solids together with tiny or high cross sections depending on the frequency of light and spacing of the materials. In crystals and glasses the optical frequencies have small probability of interaction.
x rays find most materials transparent because the photons' energy is much larger than the energies available by the fields holding the crystals together, and the scattering angles with the fields are very small, except when they hit the atoms, and then we get x ray crystallography.
Edit after comment
Here is the sequence as I see it:
A classical electromagnetic wave is made up of photons in phase according to the wave description.
There is an enormous number of photons in the wave per second making it up. Here is a useful article which explains how a classical wave is built up from a quantum substrate.
Each photon does not change the atomic or crystalline energy levels going through a transparent material in the quantum mechanical way by emitting a softer photon. It scatters quantum mechanically elastically, through the medium, changing the direction infinitesimally so that it keeps the quantum mechanical phase with its companions and displays transparency. Since a medium has a composite collective electric and magnetic field it is not a simple "electron photon going to electron photon" QED diagram. In the case of a crystal one could have a model of "photon crystal photon crystal" scattering amplitude for example.
The higher the sequential probability of scattering going through a medium the larger the final deflection through it will be, and the higher the over all probability of losing the phase with its companions in the wave.( the thicker the glass the less transparency and image coherence).
The transparency of the medium depends on the ordering of the atoms and molecules composing it so that it allows to keep the coherence between individual photons of the beam. The lower the density the better chance to keep the transparency, viz water and air.
hope this helps conceptually.
This is one of the places where wave particle dualism gets some people in trouble. Many are taught that it means that light can be a wave and a particle, and that phrasing can lead to some confusion. I find it more intuitive to just rip the bandaid off quickly and say light is neither a wave nor a particle. It is something which, in some situations, can be well modeled as a wave, and in some situations can be well modeled as a particle, but it is its own thing (which can be well modeled in all known cases using a more complicated concept, a "wavefunction").
You can think of photons getting randomly reflected or transmitted on the boundary, but the truth is that the billiard-ball photon model really isn't very effective at describing what happens at this boundary. This is one of the regions where wave mechanics models the effects very well, while particle models don't do so well. If you use wave mechanics, the idea of a wave getting partially reflected and partially transmitted isn't difficult to believe at all. In fact, it's pretty easy to prove.
Thinking in wave terms at these boundaries also gives correct answers in peculiar situations where the particle model simply falls on its face. Consider the interesting case of an "evanescent wave."
In this setup, the laser and prism are set up at the correct angles to cause "total internal reflection." This means that, by the simple models, 100% of the light should bounce off the side of the prism and into the detector. Indeed, if the prism is in the open air, we do see 100% reflection (well, within the error bars of absorption). However, bring an object close to the prisim (but not touching) and things change. You end up seeing effects from the object, even though 100% of the light was supposed to be reflected!
If you think of light like photons, this is hard to explain. If you look at it as a wave governed by Maxwell's equations, you see that you would violate the law of conservation of energy if there was a "pure" reflection. Instead it creates a reflection and an "evanescent wave" which is outside the prism, and its strength falls off exponentially, which is really hard to explain with particles!
Of course, these too are all simplifications. The real answer to your question is that the wavefunction of the light interacts with the electromagnetic fields of the atoms in the prisim, and the result of that interaction leads to reflection, refraction, diffusion, absorption, and eveansecent waves. However, naturally those equations are a bit harder to understand, so we use the older, simpler models from before quantum mechanics. We just have to be sure to use the one which is most applicable in any given situation, because none of them are quite right.
Best Answer
That's a good question. Without realising it you have stumbled across the Huygens-Fresnel principle.
The starting point it that a single silver atom is far smaller than the wavelength of light, so any scattering from it will be isotropic i.e. it will scatter the light equally in all directions.
But suppose we have two silver atoms side by side. Each atom will scatter isotropically, so in effect we have two closely spaced emitters of light and the system behaves like a Young's slits setup. Now the light isn't simply isotropically scattered, but instead it's scattered into preferred directions. (I'm oversimplifying because two atoms would be too closely spaced to act as Young's slits, but bear with me.)
Now add lots of atoms in a row, and you get something like a diffraction grating. Add lots more to make a 2D surface, then add more layers of silver atoms below, and you're building up a system where the overall light scattering is the sum of individual scattering from huge numbers of individual silver atoms. This is basically the Huygen's construction, and if you do the sums for a surface you can show that the overall scattering is only non-zero when the angle of reflection is equal to the angle of incidence. Any optics textbook should have the calculation, or a quick Google found an example here.