\ldots is using \mathellipsis for the dots. It could be redefined to use the text version, but the mixup of text and math fonts might not always a good idea.
\mathellipsis itself uses three dots as punctuation characters. That means, there is additional thin space between the dots. This can be changed by putting them into a subformula, then they are treated as \mathord atoms without additional space.
\documentclass[12pt]{article}
\usepackage{amsmath}
\renewcommand*{\mathellipsis}{%
\mathinner{{\ldotp}{\ldotp}{\ldotp}}%
}
\begin{document}
$1,2,\ldots,3$
$(1,2,\ldots)$
\end{document}
Smaller spaces
Usually thin space is the smallest space in math mode, but to some degree \ldots can be adopted to smaller spaces:
\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{letltxmacro}
\renewcommand*{\mathellipsis}{%
\mathinner{{\ldotp}{\ldotp}{\ldotp}}%
}
\makeatletter
\@ifdefinable{\org@ldots}{%
\LetLtxMacro\org@ldots\ldots
\DeclareRobustCommand*{\ldots}{%
\ifmmode
\expandafter\my@ldots
\else
\expandafter\textellipsis
\fi
}%
}
\newcommand*{\neghalfmskip}{%
\nonscript\mskip-.5\muexpr\thinmuskip\relax%
}
\newcommand*{\my@ldots}{%
\mathellipsis
\@ifnextchar,\neghalfmskip{%
\@ifnextchar:\neghalfmskip{%
\@ifnextchar;\neghalfmskip{%
\@ifnextchar.\neghalfmskip{%
\@ifnextchar!\neghalfmskip{%
\@ifnextchar?\neghalfmskip{%
\@ifnextchar){\mskip-.5\muexpr\thinmuskip\relax}{% negative kerning
}}}}}}}%
}
\makeatother
\begin{document}
\noindent
$1,2,\ldots,3_{1,2,\ldots,3}$ (ldots)\\
$1,2,\mathellipsis,3_{1,2,\mathellipsis,3}$ (mathellipsis)\\
$(1,2,\ldots)_{(1,2,\ldots)}$ (ldots)\\
$(1,2,\mathellipsis)_{(1,2,\mathellipsis)}$ (mathellipsis)
\end{document}
TeX sets a thin space between inner atoms and punctuation chars in display and text style only, therefore the use of \nonscript. There is no space in case of the closing ), but because of the character shape, you probably want to have a negative kerning, this is applied in all math styles.
More generalized detection of a closing delimiter
The example uses Andrew's hint of \rightdelim@:
\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{letltxmacro}
\renewcommand*{\mathellipsis}{%
\mathinner{{\ldotp}{\ldotp}{\ldotp}}%
}
\makeatletter
\@ifdefinable{\org@ldots}{%
\LetLtxMacro\org@ldots\ldots
\DeclareRobustCommand*{\ldots}{%
\ifmmode
\expandafter\my@ldots
\else
\expandafter\textellipsis
\fi
}%
}
\newcommand*{\neghalfmskip}{%
\nonscript\mskip-.5\muexpr\thinmuskip\relax%
}
\newcommand*{\my@ldots}{%
\mathellipsis
\@ifnextchar,\neghalfmskip{%
\@ifnextchar:\neghalfmskip{%
\@ifnextchar;\neghalfmskip{%
\@ifnextchar.\neghalfmskip{%
\@ifnextchar!\neghalfmskip{%
\@ifnextchar?\neghalfmskip{%
\rightdelim@
\ifgtest@
\mskip-.5\muexpr\thinmuskip\relax% negative kerning
\fi
}}}}}}%
}
\makeatother
\begin{document}
\noindent
$1,2,\ldots,3_{1,2,\ldots,3}$ (ldots)\\
$1,2,\mathellipsis,3_{1,2,\mathellipsis,3}$ (mathellipsis)\\
$(\neghalfmskip\ldots)_{(\neghalfmskip\ldots)}$ (neghalfmskip + ldots)\\
$(\ldots)_{(\ldots)}$ (ldots)\\
$(\mathellipsis)_{(\mathellipsis)}$ (mathellipsis)\\
$\{\ldots\}_{\{\ldots\}}$ (ldots)\\
$\{\mathellipsis\}_{\{\mathellipsis\}}$ (mathellipsis)
\end{document}
But for the detection of an opening delimiter before I do not see a way. There isn't a "\lastmathatom". Manually the space correction can be applied as the example file shows with \neghalfmskip.
in latex.ltx, \ldots is equated (\let) to \dots. so in text, it's the same, and you can use either.
with amsmath, the situation is more complicated; \dots tries to determine by context (only in math mode) whether to use \ldots or \cdots, with several more options for the author if amsmath doesn't get it right.
so in general, it's easier to just use \dots and not have to remember the details. but if the alternative chosen by amsmath isn't appropriate, the original commands can still be used explicitly.
Best Answer
The command
\ldots, which works both in text mode and in math mode, will create a single glyph in xelatex and lualatex, assuming you're using a font that knows how to set character U+2026, "Horizontal Ellipsis."In an earlier version of this answer, I mistakenly referred to what's created in pdflatex by the
\ldotscommand as a "single glyph". In fact, it's composed of three separate dots, concatenated by non-breaking whitespace. Thus, while it's not a single glyph in the narrow, technical sense, it behaves like one for typesetting purposes. For instance, it won't ever get broken up across two lines.