As a graduate student of pure mathematics, I question how valid the statement that a proper appreciation of pure math requires some knowledge of applied mathematics and theoretical physics. Perhaps part of this comes from: where is the line that separates applied mathematics and pure mathematics? If we include the whole of calculus within applied mathematics, than this is without a doubt true. But I do a lot of work in analytic number theory, and I have a hard time coming up with theoretical physics that I employ. I also have a hard time separating theoretical physics from pure mathematics - both are largely mystical, loosely formatted, and open. One might argue that theoretical physicists concern themselves more with descriptions of the natural world while mathematicians only concern themselves with what is consistent rather than what is possible... but I don't know how I feel about that either.
I will say, however, that I think both applied math and theoretical physics rely heavily on 'pure math.' I think of fields such as Lie Theory, which I consider a pure math. Lie Groups are one of the fundamental tools used in theoretical quantum physics these days - the existence of many particles and symmetries is often suspected because of mathematics and thoughts birthed within Lie Algebras.
But perhaps it is nice to know the awesome power of pure mathematics sometimes. When learning about Lie Algebra, or any sort of Abstract Algebra, I think it might be very enriching to learn about the solutions to the quantum harmonic oscillator and/or particle in a box situations. While these can be done analytically (or through divine inspiration, as seems to be what my old professor expected of us), Dirac employed a very cool use of algebra to solve these things. This included group-like behavior and the creation of 'ladder operators,' and more can be found at wikipedia. In this sense, I do appreciate pure mathematics more because of this knowledge of applied mathematics.
A large amount of group theory can also be applied to quantum mechanics. The astounding properties of the Pauli spin matrices might seem flukelike, but they can be predicted and analyzed from a group-theoretic context as well. I suspect many find is vastly satisfying to be able to predict experimental results from entirely theoretical pursuits.
As a very pure mathematician, one of the questions I am often asked is 'Why do people care about what you do?' While I might come up with something good to say, I think it is reasonable to say that the majority of the work that pure mathematicians do will never find an application that people would declare 'useful.' Thus whenever a bit of pure mathematics is 'applied,' this re-convinces people that funding the study of pure mathematics is a worthwhile endeavor. I rely on that belief for my own funding, so in that I appreciate applied mathematics a whole lot :P.
I will end with one more example. I again refer to Dirac, because I happen to know a lot about his life and how he went about his research. Dirac championed the use of projective geometry (and what I think we would now classify as differential geometry) to discover physics. Here is a transcript of a talk he gave about this subject at one time. Geometry is an interesting thing, because it's often visually based or well-grounded on intuition. Although I am not a geometer, I nonetheless am very pleased whenever I can use a physical situation (even quantum physical, slightly less intuitive) to better interpret some sort of geometric situation. Similarly, it is nice to be able to apply a geometric intuition to an apparently non-geometric problem from the applied sciences. I really encourage a quick glance through the transcript.
This is what I have for now.
------Post Edit------
I happened across a few links that I think talk about this subject nicely. One is a site that addresses how physics does not come from math alone - I take this as an example of how math is too general (in general) for physics. The second is a paper that talks about three very high-level math things that arose out of physical interpretations. A nice addendum, I hope.
There is a result in functional analysis whose first known proof uses non-standard techniques:
Theorem If a bounded linear operator $ T $ on a Hilbert space $ \mathcal{H} $ is polynomially compact, i.e., $ P(T) $ is compact for some non-zero polynomial $ P $, then $ T $ has an invariant subspace. This means that there is a non-trivial proper subspace $ W $ of $ \mathcal{H} $ such that $ p(T)[W] \subseteq W $.
The proof was given by Allen Bernstein and Abraham Robinson. Their result is significant because it is related to the so-called Invariant-Subspace Conjecture, an important unsolved problem in functional analysis. Paul Halmos, a staunch critic of non-standard analysis, supplied a standard proof of the result almost immediately after reading the pre-print of the Bernstein-Robinson paper. In fact, both proofs were published in the same issue of the Pacific Journal of Mathematics!
Best Answer
The topic of Probabilistic Combinatorics. See http://en.wikipedia.org/wiki/Probabilistic_method
This is a powerful way of giving non-constuctive existence proofs for lots of different (finite) mathematical structures and determining their properties.
I found this the highlight of all my undergraduate probability courses. If you are interested in learning more I would recommend: "The Probabilistic Method" by Alon and Spencer.