[Tex/LaTex] Typesetting the square of a mathematical operator

amsmathmath-modemath-operators

Assume I want to typeset the square of some mathematical operator A. Using \operatorname (amsmath package), there are basically two ways to do that:

  1. \(\operatorname{A^{2}}\) (i.e., the exponent is considered part of the operator name)
  2. \(\operatorname{A}^{2}\) (i.e., the exponent is not considered part of the operator name)

According to my tests, the above two formulae are not equivalent, though. In fact, the first formula has a smaller height than the second one. Minimal example:

\documentclass{article}

\usepackage{amsmath}

\newlength{\len}

\begin{document}

\begin{enumerate}
\item
  \settoheight{\len}{\(\operatorname{A^{2}}\)}
  \(\operatorname{A^{2}}\): height = \the\len
\item
  \settoheight{\len}{\(\operatorname{A}^{2}\)}
  \(\operatorname{A}^{2}\): height = \the\len
\end{enumerate}

\end{document}

Can anybody explain what is at the bottom of my observation?

Best Answer

The reason for the difference is that TeX typesets superscripts differently whether it follows a character or a box, as described in rule 18a of Appendix G of The TeXbook. As the macro \operatorname boxes its contents (because it calls \mathop which does), that's why \operatorname{A}^2 and \operatorname{A^2} differ (the first superscript concerns a box, whereas the second only the preceding A). You can easily see that an \operatorname and an \hbox behave similarly:

result of the code

\documentclass{article}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{xcolor}

\begin{document}

\begin{tabular}{ccc}
\scalebox{5}{$\operatorname{A}^2$} & \scalebox{5}{$\hbox{A}^2$} & \scalebox{5}{$\operatorname{A^2}$} \\
\verb"$\operatorname{A}^2$" & \verb"$\hbox{A}^2$" & \verb"$\operatorname{A^2}$" \\
\end{tabular}

\raisebox{1.22cm}[0pt]{\color{red}\rule{\textwidth}{0.4pt}}

\end{document}

Here are the technical details of the actual computations made by TeX in the present case:

result of the code

\documentclass[a4paper]{article}

\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{geometry}

\begin{document}

\setbox0=\hbox{$a$}% to initialize the maths fonts

\begingroup
\newdimen\h
\newdimen\q
\newdimen\boxedu
\newdimen\unboxedu
\newdimen\sigmafourteen
\newdimen\sigmafive
\q=\the\fontdimen18\scriptfont2
\sigmafourteen=\the\fontdimen14\textfont2
\sigmafive=\the\fontdimen5\textfont2
\def\tabularheading{\itshape\color{red!70!black}}

\noindent List of relevant font parameters and their values:
\begin{quote}
\begin{tabular}{lll}
\tabularheading Name & \tabularheading Symbol & \tabularheading Value \\
\texttt{x\_height} & $\sigma_5$ & \the\sigmafive \\
\texttt{sup2}      & $\sigma_{14}$ & \the\sigmafourteen \\
\texttt{sup\_drop} & $q$ (it's $\sigma_{18}$ of superscript font) & \the\q \\
\end{tabular}
\end{quote}
Comparison of the amount the superscript is shifted up for a boxed and unboxed $A$:
\begin{quote}
\setbox0=\hbox{$A$}
\h=\the\ht0
\def\maxof#1#2{%
  \ifdim#1>#2%
      #1%
    \else
      #2%
    \fi}
\begin{tabular}{lll}
& \tabularheading Boxed $A$ & \tabularheading Unboxed $A$ \\
\tabularheading height $h$ & \the\h & \the\h \\
\tabularheading base superscript shift $u_0$ & $h-q = \mathrm{\the\dimexpr\h-\q\relax}$ & 0pt \\
\tabularheading real shift $u = \max(u_0,\sigma_{14},\frac{1}{4}\sigma_5)$ &
\boxedu=\dimexpr\h-\q\relax
\boxedu=\maxof{\boxedu}{\sigmafourteen}%
\global\boxedu=\maxof{\boxedu}{.25\sigmafive}%
\the\boxedu
&
\unboxedu=0pt
\unboxedu=\maxof{\unboxedu}{\sigmafourteen}%
\global\unboxedu=\maxof{\unboxedu}{.25\sigmafive}%
\the\unboxedu
\end{tabular}
\end{quote}
Comparision of the calculations with the real typesetting:
\begin{quote}
\begin{tabular}{cc}
\scalebox{5}{$\hbox{$A$}^2$\hbox{$A$\raise\boxedu\hbox{$\scriptstyle2$}}} & \scalebox{5}{$A^2$\hbox{$A$\raise\unboxedu\hbox{$\scriptstyle2$}}} \\
\tabularheading boxed $A$ & \tabularheading unboxed $A$ \\
\end{tabular}

\raisebox{1.35cm}[0pt]{\color{blue}\rule{9.5cm}{0.4pt}}
\end{quote}
\endgroup

\end{document}
Related Question