At the start of an alignment cell TeX expands macros looking for \omit (\multicolumn) the math start in the cell template has not been added at this point so \ifmmode is always false.
You need to put \relax in front of it to kill the pre-scan for \omit or make the command containing this construct be robust so it doesn't expand in this context (using LaTeX \DeclareRobustCommand or eTeX \protected)
To show the various approaches, this plain eTeX file
\def\foo{\ifmmode \phi\else fi\fi}
\protected\def\pfoo{\ifmmode \phi\else fi\fi}
\def\rfoo{\relax\ifmmode \phi\else fi\fi}
\halign{#)&$#$\hfill&\quad dif#cult\cr
0&\omit $\phi$\hfill&\omit \quad dif{f}icult\cr
1&\omit $\phi$\hfill&\omit \quad difficult\cr
2&\foo&\foo\cr
3&\pfoo&\pfoo\cr
4&\rfoo&\rfoo\cr
}
\bye
produces
(1) is what you want.
(0) loses the ffi ligature.
(2) the non robust unprotected command gets the ffi ligature but messes up the if math test.
(3) the etex \protected macro does the same as (1).
(4) using \relax (which would be the same as using \DeclareRobustCommand in LaTeX in this context) gets the math test correct but loses the ffi ligature.
The definition of \textbackslash when the OT1 encoding is used is
\OMS-cmd \textbackslash \OMS\textbackslash
Both the first and the third token can't be typed by a user without some devious trick. However, the definition of \OMS-cmd is equivalent to
\def\OMS-cmd#1#2{%
\ifx\protect\@typeset@protect
\@inmathwarn #1%
\expandafter\ifx\csname\cf@encoding\string#1\endcsname\relax
...<omitted irrelevant things>...
}
The macro \@inmathwarn is responsible for the warning
Command \textbackslash invalid in math mode
After that the macro \OMS\textbackslash is expanded. If we were in text mode, the command would do a font change in a group, selecting a font in the OMS encoding (math symbols), doing
\char"6E
and a backslash would appear, because that's what an OMS encoded font has at slot "6E (hexadecimal).
If we're in math mode, the same \char"6E instruction would be performed, but here the font change has no effect. So we get the character at position "6E in the font in math family 0, where an n is found. When \char"6E is found in math mode, TeX does as if it were \mathchar"006E and in family 0, slot "6E there's an n (it's the upright text font and the ASCII code of n is exactly "6E).
Things are different in the T1 encoding, because in this case the definition of \textbackslash is
\T1-cmd \textbackslash \T1\textbackslash
and the latter macro eventually expands to \char"5C. In a text font, T1 encoding, at slot "5C there's actually a backslash.
Here's the explanation. Now, some advice:
Never underestimate warnings and
never use \text... symbol commands in math mode (don't mistake them with \textrm, \textit or similar, which are obviously legal in math mode).
Best Answer
amssymbdefinesemptyset