I found this post in the last few days while searching for an answer to the same problem. I'm also using XeLaTeX, and also need fontspec, so I feel your pain. I tried all kinds of things, which I won't list here.
Here's the solution I came up with:
As you discovered, the Mapping=tex-text option is what is changing the straight quotes to curly (no matter what). As you also saw, removing it results in no curly quotes anywhere. That's no good either.
The standard XeLaTeX template in TeXShop (I've excluded the sans and mono font declarations) has:
\usepackage{fontspec,xltxtra,xunicode}
\defaultfontfeatures{Mapping=tex-text}
\setmainfont[Mapping=tex-text]{Hoefler Text}
I decided to define a new font, using the same typeface as the roman font, but not include any mapping.
\newfontfamily{\S}{Hoefler Text}
As a final step, the \defaultfontfeatures{Mapping=tex-text} line must be deleted or commented out. If not, it adds mapping to all font declarations. I suppose that's meant to be helpful in case one forgets it, but in this case, it's holding us back. Since \setromanfont above already declared its mapping, removing it as a default feature won't alter the appearance of normal typing, giving you access to curly quotes.
The altered preamble section will then be:
\usepackage{fontspec,xltxtra,xunicode}
%\defaultfontfeatures{Mapping=tex-text}
\setmainfont[Mapping=tex-text]{Hoefler Text}
\newfontfamily{\S}{Hoefler Text}
When you need straight quotes, put them within the custom font call. Since there is no mapping, you will get finally them.
The ``antique'' table is 5{\S '}6{\S "} long.
I set custom commands to make writing easier.
\newcommand{\inch}{{\S "}}
\newcommand{\feet}{{\S '}}
This allows me to write:
The ``antique'' table is 5\feet 6\inch\ long.
I hope this helps. I was creating a 100+ page document, and needed both curly and straight quotes frequently. This is the only thing I've tried that works, period. If there is a more elegant way (in XeLaTeX with fontspec), I'd love to learn it. I've only been using TeX/XeLaTeX for 2-3 months, so I'm kind of a newbie.
(Minor update, April 2022: loading the textcomp package to access the glyph generated by \textquotesingle is not necessary if the vintage of your TeX distribution is more recent than 2019.)
You write
I ... just [want to] get the occasional straight quote.
The typographically correct marks for "feet" and "inches" are not (single or double, resp.) straight quote marks, but angled quote marks. These may be produced in "normal LaTeX" via $'$ and $''$, resp.
Addendum, prompted by the OP's comment that interest lies only in "straight" (vertical) quotes. In addition to the "single-quote" command \textquotesingle (requires loading the textcomp package -- if the vintage of your TeX distribution is older than 2020), there's also the \textquotedbl macro, which is available as long as a font encoding other than the original TeX font encoding (aka OT1) is used.
Here, then, is a quick MWE. Note that the text font is TNR (Times New Roman). If your publisher wants to use a font encoding other than T1 -- which is what I use in the MWE -- that's no problem at all, as long as your publisher's template doesn't impose OT1...
\documentclass{article}
% access \textquotedbl:
\usepackage[T1]{fontenc}
% access \textquotesingle:
\usepackage{textcomp} % not needed for TeX vintages more recent than 2019)
% load "Times New Roman" text font:
\usepackage{mathptmx} % (note: the "times" package is obsolete!)
\begin{document}
He exclaimed, \textquotedbl Hello,
\textquotesingle Stranger\textquotesingle.\textquotedbl
\end{document}
Second Addendum, to address a late comment by @FlashSheridan, who claims that
\textquotesingle (with or without textcomp) produces a curly closing quote, not the straight quote I need for a short C fragment.
This claim simply cannot be correct in general. As \textquotesingle is defined in the textcomp package but not the LaTeX kernel, \textquotesingle without textcomp produces an error message, not a curly closing quote. (Update April 2022: This paragraph is irrelevant if your TeX distribution is more recent than 2019.)
The following screenshots (first for Latin Modern, then for Times Roman) demonstrate conclusively that \textquotesingle does produce a straight vertical quote. In contrast, \textsf{'} does not produce a straight vertical quote -- unless, of course, some sans-serif font with straight single quote glyphs has been loaded.
Finally, the code to produce the preceding screenshots:
\documentclass[border=1pt,preview]{standalone}
\usepackage[T1]{fontenc}
\usepackage{textcomp} % for \textquotesingle macro
\usepackage{mathptmx} % or: lmodern
\begin{document}
Times Roman
\verb+\textquotesingle+: \textquotesingle
\verb+\textsf{'}+: \textsf{'}
\end{document}
Best Answer
You can have the straight quotes, provided you use some tricks.