While a definitive answer can only come from the Stanford team involved in development of TeX, and from Professor Knuth in particular, I think we can see some possible reasons.
First, Knuth designed TeX primarily to solve a particular problem (typesetting The Art of Computer Programming). He made TeX sufficiently powerful to solve the typesetting problems he faced, plus the more general case he decided to address. However, he also kept TeX (almost) as simple as necessary to achieve this. While expandable macros are useful, they are not required to solve many issues.
Secondly, there are cases where an expandable approach would be at least potentially ambiguous. Bruno's \edef\foo{\def\foo{abc}} is a good case. I'd say that here the expected result with an expandable \def is that \foo expands to nothing, but I'd also say this is not totally clear. There is the much more common case where you want something like
\begingroup
\edef\x{%
\endgroup
\def\noexpand\foo{\csname some-macro-to-fully-expand\endcsname}%
}
\x
which would be made more complex with expandable primitives.
The above example points to another grey area: what would happen about things like \begingroup and more importantly \relax. The fact that the later is a non-expandable no-op is often important in TeX programming. (Indeed, the fact that \numexpr, etc., gobble an optional trailing \relax is sometimes regarded as a bad thing.)
Finally, I suspect that ease of implementation is important. The approach of having separate expansion and execution steps makes the flow relatively easy to understand, and I also suspect to implement. An approach which mixes expansion and execution requires a more complex architecture. Here, we have to remember when Knuth was writing TeX, and the fact that programming ideas which we take for granted today were not necessarily applicable in the late 1970s. A fully-expandable approach would I suspect have made the code more complex and slower. The speed impact is one that was important when TeX was running on 'big' computers.
Here is an alternative to your current situation - using an optional argument to specify the derivative order. This way you don't have to define a macro for "each" derivative:
\deriv[<order>]{<func>}{<var>}
Here's a minimal example:
\documentclass{article}
\newcommand{\deriv}[3][]{% \deriv[<order>]{<func>}{<var>}
\ensuremath{\frac{\partial^{#1} {#2}}{\partial {#3}^{#1}}}}
\begin{document}
In text mode there is~\deriv{y}{x} and~\deriv[2]{y}{x}. In display mode there is
\[
\deriv{y}{x}\ \textrm{and}\ \deriv[2]{y}{x}\rlap{.}
\]
\end{document}
The default <order> is empty, implying the first order partial derivative. If you want the default to be 2, modify the definition to read
\newcommand{\deriv}[3][2]{...}
Technically it is possible to use a macro with numbers in them, but the usage is much less intuitive than adding something like an optional argument (as given above). Here's an implementation that now allows you to use \nameuse{deriv2}{y}{x}:
\expandafter\def\csname deriv2\endcsname#1#2{%
\ensuremath{\frac{\partial^2 {#1}}{\partial {#2}^2}}}
\makeatletter
\newcommand{\nameuse}[1]{\@nameuse{#1}}%
\makeatother
The optional argument beats this hands down.
Best Answer
As egreg explains, a fully robust check for optional arguments must use
\futureletand cannot be expandable (in fact,\newcommanddoes not make a fully correct check: try\let\lbrack[\testopt\lbrack...with your definition of\testopt, but it is possible to fix it).Expandably, there are two possibilities to look ahead: either grab an undelimited argument, or grab a delimited argument. The first one removes braces:
\testopt{[}a]{b}will be misrecognized as identical to\testopt[a]{b}. For the second method we need to decide until what tokens we should grab, and those tokens must be present. One possibility, since the second argument of\testoptis mandatory, is to force the user to put it in braces, and grab until the first open brace.Undelimited argument: we need a test for emptyness (
\detokenizesets all the catcodes to12or10, different from the catcode of$). Then define\testoptto grab one argument, and compare it with[: if it does not contain[, then there was no optional argument, and#1is the mandatory argument. Otherwise, test if it is alone in#1(slightly dirty code to cater for the case\testopt{{}[}), in which case we consider that there was an optional argument.Delimited argument: grab until a left brace, test if that starts with a
[.