On the subject of categorical versus set-theoretic foundations there
is too much complicated discussion about structure that misses the
essential point about whether "collections" are necessary.
It doesn't matter exactly what your personal list of mathematical
requirements may be -- rings, the category of them, fibrations,
2-categories or whatever -- developing the appropriate foundational
system for it is just a matter of "programming", once you understand
the general setting.
The crucial issue is whether you are taken in by the Great Set-Theoretic
Swindle that mathematics depends on collections (completed infinities).
(I am sorry that it is necessary to use strong language here in order to
flag the fact that I reject a widely held but mistaken opinion.)
Set theory as a purported foundation for mathematics does not and cannot
turn collections into objects. It just axiomatises some of the intuitions
about how we would like to handle collections, based on the relationship
called "inhabits" (eg "Paul inhabits London", "3 inhabits N"). This
binary relation, written $\epsilon$, is formalised using first order
predicate calculus, usually with just one sort, the universe of sets.
The familiar axioms of (whichever) set theory are formulae in first order
predicate calculus together with $\epsilon$.
(There are better and more modern ways of capturing the intuitions about
collections, based on the whole of the 20th century's experience of algebra
and other subjects, for example using pretoposes and arithmetic universes,
but they would be a technical distraction from the main foundational issue.)
Lawvere's "Elementary Theory of the Category of Sets" axiomatises some
of the intuitions about the category of sets, using the same methodology.
Now there are two sorts (the members of one are called "objects" or "sets"
and of the other "morphisms" or "functions"). The axioms of a category
or of an elementary topos are formulae in first order predicate calculus
together with domain, codomain, identity and composition.
Set theorists claim that this use of category theory for foundations
depends on prior use of set theory, on the grounds that you need to start
with "the collection of objects" and "the collection of morphisms".
Curiously, they think that their own approach is immune to the same
criticism.
I would like to make it clear that I do NOT share this view of Lawvere's.
Prior to 1870 completed infinities were considered to be nonsense.
When you learned arithmetic at primary school, you learned some rules that
said that, when you had certain symbols on the page in front of you,
such as "5+7", you could add certain other symbols, in this case "=12".
If you followed the rules correctly, the teacher gave you a gold star,
but if you broke them you were told off.
Maybe you learned another set of rules about how you could add lines and
circles to a geometrical figure ("Euclidean geometry"). Or another one
involving "integration by parts". And so on. NEVER was there a "completed
infinity".
Whilst the mainstream of pure mathematics allowed itself to be seduced
by completed infinities in set theory, symbolic logic continued and
continues to formulate systems of rules that permit certain additions
to be made to arrays of characters written on a page. There are many
different systems -- the point of my opening paragraph is that you can
design your own system to meet your own mathematical requirements --
but a certain degree of uniformity has been achieved in the way that they
are presented.
We need an inexhaustible supply of VARIABLES for which we may substitute.
There are FUNCTION SYMBOLS that form terms from variables and other terms.
There are BASE TYPES such as 0 and N, and CONSTRUCTORS for forming new
types, such as $\times$, $+$, $/$, $\to$, ....
There are TRUTH VALUES ($\bot$ and $\top$), RELATION SYMBOLS ($=$)
and CONNECTIVES and QUANTIFIERS for forming new predicates.
Each variable has a type, formation of terms and predicates must respect
certain typing rules, and each formation, equality or assertion of a
predicate is made in the CONTEXT of certain type-assignments and
assumptions.
There are RULES for asserting equations, predicates, etc.
We can, for example, formulate ZERMELO TYPE THEORY in this style. It has
type-constructors called powerset and {x:X|p(x)} and a relation-symbol
called $\epsilon$. Obviously I am not going to write out all of the details
here, but it is not difficult to make this agree with what ordinary
mathematicians call "set theory" and is adequate for most of their
requirements
Alternatively, one can formulate the theory of an elementary topos is this
style, or any other categorical structure that you require. Then a "ring"
is a type together with some morphisms for which certain equations are
provable.
If you want to talk about "the category of sets" or "the category of rings"
WITHIN your tpe theory then this can be done by adding types known as
"universes", terms that give names to objects in the internal category
of sets and a dependent type that provides a way of externalising
the internal sets.
So, although the methodology is the one that is practised by type theorists,
it can equally well be used for category theory and the traditional purposes
of pure mathematics. (In fact, it is better to formalise a type theory
such as my "Zermelo type theory" and then use a uniform construction to
turn it into a category such as a topos. This is easier because the
associativity of composition is awkward to handle in a recursive setting.
However, this is a technical footnote.)
A lot of these ideas are covered in my book "Practical Foundations of
Mathematics" (CUP 1999), http://www.PaulTaylor.EU/Practical-Foundations
Since writing the book I have written things in a more type-theoretic
than categorical style, but they are equivalent. My programme called
"Abstract Stone Duality", http://www.PaulTaylor.EU/ASD is an example of the
methodology above, but far more radical than the context of this question
in its rejection of set theory, ie I see toposes as being just as bad.
(It happens that I will be giving a talk this week on this topic at the Exploring the Frontiers of Incompleteness series at Harvard, which is focussing on the question of determinism in set theory. The materials for all the talks are posted on the EFI web page, which also includes videos and discussion forums.)
Set theory currently provides at least a robust informal treatment of the multiverse. What I mean by this is that it is part of standard set theoretic practice and understanding to point out when a set-theoretic assertion $\varphi$ can have different truth values in different set-theoretic universes. That is, set theorists are already paying attention to the multiverse, when they discuss such issues as independence, forcing absoluteness, indestructibility and so on, which all concern the issue of how set-theoretic assertions can vary in the mutliverse.
Another more formal treatment of the multiverse idea is provided by what I have described as the toy model formalization, where one considers a multiverse from the perspective of a much larger set-theoretic background. For example, in ZFC one may consider the multiverse of all countable models of set theory, or just of a portion of them. This is the approach that Gitman and I took in our consistency proof that you mentioned, A natural model of the multiverse axioms. One can also view the typical countable-transitive-model approach to forcing as an instance of the toy model formalization.
Perhaps the toy model formalization has some affinity with the idea of the axiom of universes in category theory, and such a formalization would lead to a tower of multiverse concepts, one existing at the level of each Grothendieck universe. So it seems clear that one may set up a formal theory of multiverses using ZFC as a background theory, supplemented with various universe axioms, and leading to a tower of multiverses.
But my opinion is that the toy model method of formalization is ultimately unsatisfying, since it leads one to adopt axioms and principles about the toy models, rather than about the background set theory, which is the intended goal.
Although many of the questions that we have about the full multiverse happen to be first-order expressible in the language of set theory, nevertheless many of the questions that we are interested in happen to be formalizable in first-order ZFC. For example, many of the concepts of set-theoretic geology are formalizable in first-order ZFC, such as the open question whether every two ground models of the universe have a deeper ground inside them. Similarly, the questions surrounding the modal logic of forcing are also first-order expressible. This is a happy phenomenon, when we are able to use our current formalization to express and understand the multiverse issues. But the troubling matter underlying your question is that some of our questions are indeed pushing up against our formalism, which may be inadequate to the task.
Best Answer
Set theory provides a foundation for mathematics in roughly the same way that Turing machines provide a foundation for computer science. A computer program written in Java or assembly language isn't actually a Turing machine, and there are lots of good reasons not to do real programming in Turing machines - real languages have all sorts of useful higher order concepts. But Turing machines are a useful foundation because everything else can be encoded by Turing machines, and because it's much easier to study Turing machines than it is to study a more complicated higher order language.
Similarly, the point isn't that every mathematical object is a set, the point is that every mathematical object can be encoded by a set. It doesn't represent higher level ideas, like the fact that mathematical objects usually have types (as one of my colleagues likes to point out, the question "is the integer 6 an abelian group" is technically a reasonable one in set theory, but not in mathematics). But it's a (relatively) simple system to study, and just about everything we want to do can be encoded in set theory.
To answer your specific questions, yes, it's still true that every mathematical object can be encoded as a set. Because sets are very flexible, there's no reason to think this will not continue to be true. There is no current field of mathematics in which urelements are essential, and because things one would do with urelements can instead be encoded with sets, there is unlikely to be such a field.
ZFC does impose some limitations on category theory, because it doesn't allow objects on the same scale of the universe of sets. (For instance the category of categories is awkward to consider within ZFC, because the objects of this category cannot be a set.) These are reflected in the discussions of "small" and "locally small" categories. These issues can be worked around in mild extensions of ZFC by using things like Grothendieck universes. (Note that this is a feature of ZFC, not of set theoretic foundations in general. Quine's New Foundations allows certain self-containing sets.)
This way of thinking can't really be burden because ZFC doesn't impose a way of thinking. The fact that things can be encoded as sets doesn't, and shouldn't, mean that we always think of them that way. It's perfectly consistent with having a set theoretic foundation to work with things like urelements, or to think about groups and categories without thinking of them as sets. (Worrying about things like self-containing categories can be a burden, but it's a necessary one given the history of paradoxical objects containing themselves.)