LyX 2.1 has support for new fonts. Note, in particular, the URW Garamond and Libertine additions.
LyX 2.1 is in beta 2 and is stable (we have had no serious reports of new bugs). You can easily install the development version of 2.1 in Ubuntu with the PPA without risk of conflict with your current 2.0.x version.
Otherwise, use XeTeX or LuaTeX. Your fears that LyX will have problems typesetting them are not correct. The XeTeX and LuaTeX engines do all of the work, not LyX.
I'm not sure I qualify as experienced and/or more successful :-). But here's an example of the setup I used for my thesis. Adobe Garamond Pro was used as the text font. Sadly, I needed bold to appease the thesis arbiters at my university, so I used BoldFont=AGaramondPro-Semibold as it looks slightly less out of place than the standard bold weight.
For mathematics, I used the mathdesign package with option adobe-garamond. This provided all math symbols I needed for my document, with glyph shapes appropriate for Garamond as a text font. For this to be successful, you must copy the .otf font files to a specific TDS directory, as detailed in the mathdesign README file.
I also have shown some additional typographical enhancements in my sample below, which you may be interested in. microtype allows for more pleasing color on the page. The selnolig package breaks ligatures which would otherwise span morpheme boundaries. Of course, only German and English are supported out of the box, so it won't do much for the Latin text in my example below.
Here's an example using the blindtext package to produce a sample mathematics paper, with explanations in the comments for each font option. I've set this in a two-column layout just so that all the math samples fit on one page for the screenshot.
Sample Code
%% compile with LuaLaTeX
\documentclass[twocolumn]{article}
\usepackage{blindtext} % just for the sample math paper
% here are the font specifications
\usepackage[adobe-garamond]{mathdesign} % use the mathdesign package with Garamond
\usepackage[no-math]{fontspec} % load fontspec without modifying maths fonts
\setmainfont[
Ligatures=TeX, % make ---, --, etc function as in traditional (La)TeX
Kerning=Uppercase, % slight tracking adjustment for all-caps material
BoldFont={AGaramondPro-Semibold}, % this is a bit of a faux pas, but it was required by my thesis office :(
Numbers=Proportional, % I also had some code to switch to lining numbers in tabular matter
]{Adobe Garamond Pro}
\setmathrm{Adobe Garamond Pro} % for operators and other text in math
% some additional (optional) typographic enhancements
\usepackage{microtype}
\usepackage[english]{selnolig} % of course, this won't do much for latin lipsum text :-)
\begin{document}
\blindmathpaper
\end{document}
Sample Output
Close-up
Best Answer
I think you're looking for the
ifplatformpackage.Shell escape must be enabled for this to work, but try the following:
FYI, there is also
ifxetex, which is nice for setting up fonts, encodings, etc. based on your chosen typesetting engine.