First, most mathematicians don't really care whether all sets are "pure" -- i.e., only contain sets as elements -- or not. The theoretical justification for this is that, assuming the Axiom of Choice, every set can be put in bijection with a pure set -- namely a von Neumann ordinal.
I would describe Bourbaki's approach as "structuralist", meaning that all structure is based on sets (I wouldn't take this as a philosophical position; it's the the most familiar and possibly the simplest way to set things up), but it is never fruitful to inquire as to what kind of objects the sets contain. I view this as perhaps the key point of "abstract" mathematics in the sense that the term has been used for past century or so. E.g. an abstract group is a set with a binary relation: part of what "abstract" means is that it won't help you to ask whether the elements of the group are numbers, or sets, or people, or what.
I say this without having ever read Bourbaki's volumes on Set Theory, and I claim that this somehow strengthens my position!
Namely, Bourbaki is relentlessly linear in its exposition, across thousands of pages: if you want to read about the completion of a local ring (in Commutative Algebra), you had better know about Cauchy filters on a uniform space (in General Topology). In places I feel that Bourbaki overemphasizes logical dependencies and therefore makes strange expository choices: e.g. they don't want to talk about metric spaces until they have "rigorously defined" the real numbers, and they don't want to do that until they have the theory of completion of a uniform space. This is unduly fastidious: certainly by 1900 people knew any number of ways to rigorously construct the real numbers that did not require 300 pages of preliminaries.
However, I have never in my reading of Bourbaki (I've flipped through about five of their books) been stymied by a reference back to some previous set-theoretic construction. I also learned only late in the day that the "structures" they speak of actually get a formal definition somewhere in the early volumes: again, I didn't know this because whatever "structure-preserving maps" they were talking about were always clear from the context.
Some have argued that Bourbaki's true inclinations were closer to a proto-categorical take on things. (One must remember that Bourbaki began in the 1930's, before category theory existed, and their treatment of mathematics is consciously "conservative": it's not their intention to introduce you to the latest fads.) In particular, apparently among the many unfinished books of Bourbaki lying on the shelf somewhere in Paris is one on Category Theory, written mostly by Grothendieck. The lack of explicit mention of the simplest categorical concepts is one of the things which makes their work look dated to modern eyes.
Set theory provides a foundation for mathematics in roughly the same way that Turing machines provide a foundation for computer science. A computer program written in Java or assembly language isn't actually a Turing machine, and there are lots of good reasons not to do real programming in Turing machines - real languages have all sorts of useful higher order concepts. But Turing machines are a useful foundation because everything else can be encoded by Turing machines, and because it's much easier to study Turing machines than it is to study a more complicated higher order language.
Similarly, the point isn't that every mathematical object is a set, the point is that every mathematical object can be encoded by a set. It doesn't represent higher level ideas, like the fact that mathematical objects usually have types (as one of my colleagues likes to point out, the question "is the integer 6 an abelian group" is technically a reasonable one in set theory, but not in mathematics). But it's a (relatively) simple system to study, and just about everything we want to do can be encoded in set theory.
To answer your specific questions, yes, it's still true that every mathematical object can be encoded as a set. Because sets are very flexible, there's no reason to think this will not continue to be true. There is no current field of mathematics in which urelements are essential, and because things one would do with urelements can instead be encoded with sets, there is unlikely to be such a field.
ZFC does impose some limitations on category theory, because it doesn't allow objects on the same scale of the universe of sets. (For instance the category of categories is awkward to consider within ZFC, because the objects of this category cannot be a set.) These are reflected in the discussions of "small" and "locally small" categories. These issues can be worked around in mild extensions of ZFC by using things like Grothendieck universes. (Note that this is a feature of ZFC, not of set theoretic foundations in general. Quine's New Foundations allows certain self-containing sets.)
This way of thinking can't really be burden because ZFC doesn't impose a way of thinking. The fact that things can be encoded as sets doesn't, and shouldn't, mean that we always think of them that way. It's perfectly consistent with having a set theoretic foundation to work with things like urelements, or to think about groups and categories without thinking of them as sets. (Worrying about things like self-containing categories can be a burden, but it's a necessary one given the history of paradoxical objects containing themselves.)
Best Answer
Here's an example of something that I think Mumford might advocate in the foundations of mathematics: Solovay's model.
The axiom of choice is generally accepted by mathematicians, but it has always suffered from the nagging problem that it violates certain intuitions we have. Almost all these counterintuitive consequences of the axiom of choice are related in one way or another to the existence of non-measurable sets. (See this related MO question for more information, in particular Ron Maimon's answer.) Solovay's model shows that we can come close to having our cake and eating it too: We can simultaneously have the axioms "all Lebesgue sets are measurable" and the axiom of dependent choice. The former pretty much eliminates all the probabilistic paradoxes while the latter gives us almost all of the "desirable" consequences of the axiom of choice.
The reason that I think this is the sort of thing Mumford might advocate is that Mumford's discussion of Freiling's theorem shows that he really wants to preserve probabilistic intuition even at the expense of jettisoning a well-accepted axiom. In the paper he suggests getting rid of the power-set axiom, but my guess is he was probably not familiar with Solovay's model at the time, and if he were, he would have been favorably disposed towards it.
EDIT: In particular, in Solovay's model, all the following hold: (1) the axioms of ZF, including powerset; (2) all sets are Lebesgue measurable (which is most of what we need to capture probabilistic intuitions); (3) Freiling's axiom of symmetry; (4) the continuum hypothesis in the form "every uncountable subset of $\mathbb R$ can be put into 1-1 correspondence with $\mathbb R$." The only price one pays is that the axiom of choice has to be weakened to dependent choice. (Thanks to Ali Enayat for pointing this out.) My view is that Freiling's argument shows only that probabilistic intuition is incompatible with full-blown AC (which is something we knew already); the continuum hypothesis is a red herring.
For more information about the practical impact of adopting Solovay's model and some speculation on why it hasn't already been adopted widely, see this MO question and Andreas Blass's answer to this MO question.