There is good discussion of what is the difference between Latin Modern and cm-super?
Can someone summarise what is the differences and similarities between Latin Modern, Linux Libertine and Computer Modern Unicode fonts?
All these families originate from Knuth's CM metafont fonts. All are availabe as Open Type fonts. The major difference, from my point of view, is absence of Cyrillic letters in LM. But my main interest here is typesetting quality from professional point of view: outlines, kerning, protrusion etc., i.e. those features which are not visible for unexperienced user. I understand that there are many similar questions in TeX.sx, so what I'd like to get is some sort of summary or FAQ. Important point is which of these fonts are frozen and which is actively supported.
Best Answer
Linux Libertine is complete different from the other two. LM and CM are modern types while Libertine is transitional. Most famous transitional types are Times Roman and Times New Roman. In fact Libertine aims as a opensource replacement for Times. Notable modern types are Bodoni, Didone, etc. Classification of types are arguable. But compare a typical transitional type with a modern type is still meaningless. They have very different historical origins and spirits.
I have no idea what you mean by "Outline", OpenType fonts are all outlined fonts.
Kerning is indeed important. In my personal opinion, Knuth's CM is the best. LM is good, but some part of its extension to CM still requires fine tuning. Libertine is also of good quality.
Protrusion, and expansion are set by the users and the goodness of these does not depend on the type itself but how you use these types and fine tune your microtype configurations. Currently there are negative-value protrusion bug in LuaTeX, which limits its use in fine adjustment of protrusion, especially for italic shapes, which often requires negative protrusion. Expansion requires less effort and the improvement of more sophisticated setting than defaults are quite marginal.
If you do want professional typesetting quality, I suggest you at least gain some basic type knowledge first. Typesetting Odyssey in a modern type like CM will looks wired however good the type itself. Choosing the right type for the right work is the first step.
The second step is acquire a good copy of the chosen type. For example, CM is a good type if you decided to use a modern type. LinoType's version of Times Roman is better than Monotype's Times New Roman in my opinion. When it is too expensive for the work at hand, Libertine can be used instead.
The third step is to setup sensible protrusion and expansion. This requires hard work and patience and experience. But you will get there someday.
The fourth step will be editing the font files to obtain even better quality. Fix missing kerning, etc. Fix a font file is no different than bug fix in software development, and just like every software (except Knuth's final version of TeX as far as I know), there will be bugs. (Well, read your EULA first)
As mentioned by @Patrick, kerning pairs can be added with a feature file in the case of LuaTeX.
In summary, choose and use types wisely. The technical advantages or disadvantages of certain types are less important for good typesetting. And technical issues can in many cases be fixed.