The characteristic feature of a conducting medium is its presence of free electric charges on its surface, which are unbound. Unbound, free electrons do not have a restoring force and therefore, have no natural frequencies; however, they will always oscillate at the driving frequency. When an impinging EM wave oscillates these free electrons at optical frequencies, the free electrons will re-emit light with exactly the same driving frequencies. Since all of the visible frequencies are re-emitted, the metal surface looks “silvery."
The question is a lot more interesting when we address why gold or copper is colored, not silvery. Intuitively, if the free electrons are the cause of the silvery-look, then it must be the bound electrons in, say gold, which is responsible for its golden color. What else can it be? Bound electrons do have restoring forces and therefore, they have resonant frequencies. Because bound electrons in conductors behave the same as bound electrons in dielectrics, I can use this link to model the index of refraction as
$$n^2(ω) = 1 + \frac{Ne^2}{m ϵ_0}\left[\frac{-f_e}{ω^2-iβω} + \frac{f_j}{ω_{0j}^2-ω^2+iβ_jω}\right]$$
The first bracketed term is the contribution from the free electrons (note no natural frequencies $ω_0$, so no absorption) whereas the second term arises from the bound electrons (natural frequencies $ω_{0j}$ imply absorption). So for gold to appear a reddish-yellowish color (λ ≥ 500 nm), the blues and greens (λ ≤ 500 nm) must interact more with the bound electrons. I then expect gold to have larger skin depth values for λ ≤ 500 nm and smaller ones for λ ≥ 500 nm. On the other hand, I expect silver to have smaller skin depth values for all optical frequencies since there is apparently no absorption of any wavelengths. So the key is the skin depth values.
Skin Depth Values:
The skin depth $\delta$ is determined from the absorption coefficient where $\delta = 1/\alpha$. Using the website refractiveindex.info provided by Stefan Bischof’s post, I obtained $\alpha$-values and plotted $\delta$ vs. $\lambda$ for six different metals (Au, Ag, Cu, Al, Ni and Pt) using Excel. (Note that I am not showing the origin in the plot.)
Focusing on gold and silver only, the data shows exactly what I expected. Between 400 nm ≤ λ ≤ 500 nm, gold’s skin depth is increasing with longer wavelength to just about 490 nm, implying that the blues and greens are more likely to interact with the bound electrons and get absorbed. After the 500 nm peak, the skin depth steadily decreases implying that the red end of the visible spectrum is getting re-emitted by the free electrons. Silver on the other hand, has smaller δ-values (especially around 500 nm) suggesting that there is little interaction with the bound electrons and reflects all frequencies alike. Overall, the trend is that the silvery metals (Ag, Al, Ni and Pt) all have smaller skin depths while the colored metals (Au and Cu) have larger skin depths, as expected. I believe that this suggests that among other things, penetration depth will help to determine whether bound electrons are playing a role in the color of the metal.
Many thanks to Stefan Bischof for approaching the question in the manner he did.
Several things contribute to conductivity. I think two are particularly relevant to the question. First concerns band structure. Metals can conduct easily because there are empty states very close in energy to the Fermi surface, so that it is easy to promote an electron to a higher-energy level. In an electric field, the newly-occupied states all (loosely speaking) have momenta in the direction of the field, so there is net movement of the electrons in the direction of the field. How well this works depends on the density of states at the Fermi level. The more states available, the better the conductivity. I've looked for these data for the noble metals but haven't located them.
The second concerns the dominant loss mechanism at room temperature, the interaction of the electrons with the lattice. In a simple model of a solid the positions of the ion cores are taken to be fixed. In reality they are not, and a passing electron will excite lattice excitations (phonons) and lose energy. How well this works depends on the strength of the electron-phonon coupling in the solid, and that's complicated business.
Having said all that, I don't have data at hand to make an argument about the noble metals. But I think it should be clear that your picture of conductivity needs to be augmented, and that might be enough to answer your question.
Update I found data on density of states, and the order of densities of states is Ag, Cu, Au, from highest to lowest.
Best Answer
Why do most metals appear silver in color, with gold being an exception?
It is hardly surprising that the answer to this question relies heavily on quantum theory, but most people will be surprised to hear that the full answer brings relativistic considerations into the picture. So we are talking quantum relativistic effects.
The quantum bit of the story tells us that the colour of metals such as silver and gold is a direct consequence of the absorption of photons by d electrons. This photon absorption results in d electrons jumping to s orbitals. Typically, and certainly for silver, the 4d→5s transition has a large energy separation requiring ultraviolet photons to enable the transition. Therefore, photons with frequencies in the visible band have insufficient energy to be absorbed. With all visible frequencies reflected, silver has no colour of its own: it's reflective, an appearance we refer to as 'silvery'.
Now the relativistic bit. It is important to realize that electrons in the s orbitals have a much higher likelihood of being in the neighborhood of the nucleus. Classically speaking, being close to the nucleus means higher velocities (cf speed of inner planets in solar system with that of the outer planets).
For gold (with atomic number 79 and hence a highly charged nucleus) this classical picture translates into relativistic speeds for electrons in s orbitals. As a result, a relativistic contraction applies to the s orbitals of gold, which causes their energy levels to shift closer to those of the d orbitals (which are localized away from the nucleus and classically speaking have lower speeds and therefore less affected by relativity). This shifts the light absorption (for gold primarily due to the 5d→6s transition) from the ultraviolet down to the lower frequency blue range. So gold tends to absorb blue light while it reflects the rest of the visible spectrum. This causes the yellowish hue we call 'golden'.
Reflectivity as function of wavelength. Purple/blue light corresponds to 400 - 500 nm, the red end of the visible spectrum to about 700 nm.
See: the color of gold, relativistic quantum chemistry.