[Tex/LaTex] Open Type Fonts have different scale in XeLaTeX

fontsfontsizefontspecopentype

I am trying to create a template that allows easy switching of the main font using fontspec. I have noticed that the size of different fonts varies greatly, as the picture below illustrates. If we take the Palatino fonts as the benchmark, Adobe Garamond Pro and Linux Libertine are much smaller and Minion Pro and Adobe Caslon Pro are slightly larger than those, but still smaller than Palatino.

I always thought that open type fonts are somewhat "normalised" so that this does not occur. I could now use trial and error with fontspec's scale option, but maybe there are some guidelines out there that tell me what I am supposed to do to scale fonts to the same size.

Comparison of Open Type Fonts

This has been generated with the following MWE (except Pazo Math):

\documentclass{scrartcl}
\usepackage{fontspec}
\defaultfontfeatures{Mapping=tex-text}

\begin{document}

\setmainfont{TeX Gyre Pagella}
The quick brown fox jumps over the lazy dog (TeX Gyre Pagella)

\setmainfont{Adobe Garamond Pro}
The quick brown fox jumps over the lazy dog (Adobe Garamond Pro)

\setmainfont{Linux Libertine O}
The quick brown fox jumps over the lazy dog (Linux Libertine)

\setmainfont{Adobe Caslon Pro}
The quick brown fox jumps over the lazy dog (Adobe Caslon Pro)

\setmainfont{Minion Pro}
The quick brown fox jumps over the lazy dog (Minion Pro)

\end{document}