The problem is that the \begin{...} and \end{...} pair commands automatically create a "group" so that, in effect, the & and \\ are "out of scope" for the tabular, while inside the "production" environment they just show up at a place where the compiler is not expecting them.
A second problem with your definitions is that, even if they would work, they would be adding an extra \\ at the end of the tabular, adding an unwanted space at the end. Perhaps some more appropriate definitions would be
\documentclass[a4paper,10pt]{article}
\newcommand{\production}[1]{#1 ::= &}
\newenvironment{grammar}{\tabular{p{3cm}l}}{\endtabular}
\begin{document}
\begin{grammar}
\production{XmlStartTag} ... \\
\production{XmlOtherTag} ... \\
\production{XmlEndTag} ...
\end{grammar}
\end{document}
Note no \\ at the end of the last production. Also in the definition of the grammar you don't need to repeat the work of \begin/\end, and you can instead directly use \tabular and \endtabular.
While a definitive answer can only come from the Stanford team involved in development of TeX, and from Professor Knuth in particular, I think we can see some possible reasons.
First, Knuth designed TeX primarily to solve a particular problem (typesetting The Art of Computer Programming). He made TeX sufficiently powerful to solve the typesetting problems he faced, plus the more general case he decided to address. However, he also kept TeX (almost) as simple as necessary to achieve this. While expandable macros are useful, they are not required to solve many issues.
Secondly, there are cases where an expandable approach would be at least potentially ambiguous. Bruno's \edef\foo{\def\foo{abc}} is a good case. I'd say that here the expected result with an expandable \def is that \foo expands to nothing, but I'd also say this is not totally clear. There is the much more common case where you want something like
\begingroup
\edef\x{%
\endgroup
\def\noexpand\foo{\csname some-macro-to-fully-expand\endcsname}%
}
\x
which would be made more complex with expandable primitives.
The above example points to another grey area: what would happen about things like \begingroup and more importantly \relax. The fact that the later is a non-expandable no-op is often important in TeX programming. (Indeed, the fact that \numexpr, etc., gobble an optional trailing \relax is sometimes regarded as a bad thing.)
Finally, I suspect that ease of implementation is important. The approach of having separate expansion and execution steps makes the flow relatively easy to understand, and I also suspect to implement. An approach which mixes expansion and execution requires a more complex architecture. Here, we have to remember when Knuth was writing TeX, and the fact that programming ideas which we take for granted today were not necessarily applicable in the late 1970s. A fully-expandable approach would I suspect have made the code more complex and slower. The speed impact is one that was important when TeX was running on 'big' computers.
Best Answer
You can try something like that but there are several possibilities but what do you want to do ?