A Grothendieck universe is known in set theory as the set Vκ for a (strongly) inaccessible cardinal κ. They are exactly the same thing. Thus, the existence of a Grothendieck universe is exactly equivalent to the existence of one inaccessible cardinal. These cardinals and the corresponding universes have been studied in set theory for over a century.
The Grothendieck Universe axiom (AU) is the assertion that every set is an element of a universe in this sense. Thus, it is equivalent to the assertion that the inaccessible cardinals are unbounded in the cardinals. In other words, that there is a proper class of inaccessible cardinals. This is the axiom you sought, which is exactly equivalent to AU. In this sense, the axiom AU is a statement in set theory, having nothing necessarily to do with category theory.
The large cardinal axioms are fruitfully measured in strength not just by direct implication, but also by their consistency strength. One large cardinal property LC1 is stronger than another LC2 in consistency strength if the consistency of ZFC with an LC1 large cardinal implies the consistency of ZFC with an LC2 large cardinal.
Measured in this way, the AU axiom has a stronger consistency strength than the existence of any finite or accessible number of inaccessible cardinals, and so one might think it rather strong. But actually, it is much weaker than the existence of a single Mahlo cardinal, the traditional next-step-up in the large cardinal hierarchy. The reason is that if κ is Mahlo, then κ is a limit of inaccessible cardinals, and so Vκ will satisfy ZFC plus the AU axiom. The difference between AU and Mahloness has to do with the thickness of the class of inaccessible cardinals. For example, strictly stronger than AU and weaker than a Mahlo cardinal is the assertion that the inaccessible cardinals form a stationary proper class, an assertion known as the Levy Scheme (which is provably equivconsistent with some other interesting axioms of set theory, such as the boldface Maximality Principle, which I have studied a lot). Even Mahlo cardinals are regarded as rather low in the large cardinal hierarchy, far below the weakly compact cardinals, Ramsey cardinals, measurable cardinals, strong cardinals and supercompact cardinals. In particular, if δ is any of these large cardinals, then δ is a limit of Mahlo cardinals, and certainly a limit of strongly inaccessible cardinals. So in particular, Vδ will be a model of the AU axiom.
Rather few of the large cardinal axioms imnply AU directly, since most of them remain true if one were to cut off the universe at a given inaccessible cardinal, a process that kills AU. Nevertheless, implicit beteween levels of the large caridnal hiearchy are the axioms of the same form as AU, which assert an unbounded class of the given cardinal. For example, one might want to have unboundedly many Mahlo cardinals, or unboundedly many measurable cardinals, and so on. And the consistency strength of these axioms is still below the consistency strength of a single supercompact cardinal. The hierarchy is extremely fine and intensely studied. For example, the assertion that there are unboundedly many strong cardinals is equiconsistent with the impossiblity to affect projective truth by forcing. The existence of a proper class of Woodin cardinals is particularly robust set-theoretically, and all of these axioms are far stronger than AU.
There are natural weakenings of AU that still allow for almost all if not all of what category theorists do with these universes. Namely, with the universes, it would seem to suffice for almost all category-theoretic purposes, if a given universe U were merely a model of ZFC, rather than Vκ for an inaccessible cardinal κ. The difference is that U is merely a model of the Power set axiom, rather than actually being closed under the true power sets (and similarly using Replacement in place of regularity). The weakening of AU I have in mind is the axiom that asserts that every set is an element of a transitive model of ZFC. This assertion is strictly weaker in consistency strength thatn even a single inaccessible cardinal. One can get much lower, if one weakens the concept of universe to just a fragment of ZFC. Then one could arrive at a version of AU that was actually provable in ZFC, but which could be used for most all of the applications in cateogory theory to my knowledge. In this sense, ZFC itself is a kind of large cardinal axiom relative to the weaker fragments of ZFC.
Best Answer
I do not know of any active set theorists who think large cardinals are inconsistent. At least, within the realm of cardinals we have seriously studied.
[Reinhardt suggested an ultimate axiom of the form "there is a non-trivial elementary embedding $j:V\to V$". Though some serious set theorists found it of possible interest immediately following its formulation, Kunen quickly afterward showed that this is inconsistent, using choice. It is not known whether choice is needed, but current research suggest that, even without choice, natural strengthenings of this axiom may be inconsistent in $\mathsf{ZF}$ alone. This is hardly an argument against large cardinals in general. Instead, it provides us with natural limitations on their reach. For another example, see here.]
There are several active set theorists who do not commit themselves one way or the other to the consistency of large cardinals, but use them if necessary, and do not object on mathematical grounds to arguments that involve them. For some of them, set theory is about all possible (natural) extensions of $\mathsf{ZFC}$, and there are certainly many interesting such extensions (such as $V=L$) that rule out large cardinals. Thus, it is not that they consider the cardinals inconsistent.
(I confess, this may be ignorance on my part.)
And I know of only two mathematicians who years ago were serious set theorists and who have expressed doubts about the consistency of (certain) large cardinals. Neither is currently active within the field, and so their position should be taken with a grain of salt, since it missed the significant results from the late 80s that could very well have forced them to reconsider.
Why do we expect $\mathsf{ZFC}$ to be consistent, to begin with? We expect more than mere consistency, of course, but doubting large cardinals usually means distrust in set theory as a whole. I am not a philosopher, so I will not discuss philosophical positions or justifications. A good reference for the heuristics behind the basic ZFC axioms is the wonderful paper
Large cardinals are discussed in its follow-up,
Maddy's two books on Platonism and Naturalism discuss extensively why the view of a set theoretic universe with large cardinals rather than not is the reasonable choice, given our current understanding, see
and
The books present several subtle technical points that can only be completely understood once one is aware of the deep connections between large cardinals and (generic) absoluteness. Maddy's more recent views on the subject can be seen here:
How do set theorists measure the internal plausibility of large cardinal assumptions, beyond their usefulness in proving results? The point of the inner model program (and of its most recent offspring, descriptive inner model theory) is to develop fine structural ("$L$ like") models for large cardinals. These models are canonical in several precise ways, and have a rich internal structure that many set theorists take as evidence of the consistency of the large cardinals under consideration. Thanks to its advances, we have a much clearer view of the set theoretic universe nowadays (for example, we now have the different covering lemmas, and several generic invariance results) than when the program began, motivated by what we now call Gödel's program.
The program has currently reached well past Woodin cardinals, but is not yet at the level of supercompact cardinals. This can be interpreted as saying that, using the strongest tools currently at our disposal, we are fairly certain of the consistency of, say "there is a Woodin limit of Woodin cardinals". Time will tell whether the program will reach supercompactness. If it does not, this will provide us with strong evidence of their inconsistency, though I am not sure anybody actually expects this to be the outcome.
John Steel and Tony Martin have over the years refined something they call "the speech", where they explain their position towards large cardinals. It is well worth reading, and trying to summarize it in a few lines would be an injustice. It can be found in these two postings to the Foundations of Mathematics (FOM) list: 1, 2 (the notation here is $P_T =$ set of $\Pi^0_1$ consequences of $T$), and in the papers from the "Does mathematics need new axioms?" panel, see
Steel's own views are also presented in some detail in Maddy's books. For very recent developments, see his talk:
At the risk of not being balanced, let me point out some highlights: We have a coherent picture of the universe of sets, with large cardinals. We can, within this picture, interpret theories where there are no such cardinals. However, we do not have such a coherent picture in the opposite direction. The consequences of large cardinals, at the arithmetic level (and more, as we climb up through the hierarchy) are compatible. The arithmetic consequences of any natural extension of $\mathsf{ZFC}$ fall somewhere within this hierarchy (as far as the theories we can currently analyze), even if the theory does not mention large cardinals. In fact, determinacy statements, incompatible with choice, also fall within this hierarchy and are mutually interpretable with large cardinals (again, as far as those theories we can currently analyze). This deep connections with determinacy are behind what we now call descriptive inner model theory, see
Large cardinals provide us with generic absoluteness, and generic absoluteness, a natural requirement if we are interested in understanding the projective theory of the reals, requires the consistency of large cardinals. See this answer for a bit more on this issue; let me emphasize that this is not some technical or artificial requirement, but rather a natural extension of basic results in classical descriptive set theory.
Large cardinals seem inherently necessary to mathematical practice, not just set theory. Harvey Friedman has written extensively on this issue.
In short: We have a very clear measure of progress understanding large cardinals and their consequences. By this measure, we can now understand many set theoretical issues that do not involve large cardinals but for which they are necessary in deeper ways (not just consistency-wise). This measure actually requires the large cardinals, we do not have anything like that without them. This measure is meaningful even in settings that are not set theoretical, and seems unavoidable even within mathematical practice (though it is perhaps too soon to tell how significant this will be at the end for "practicing mathematicians"). We do not have any serious mathematical model where large cardinals would be inconsistent, however, we have a serious program of research that would ultimately teach us that, were this the case. The program has provided us, instead, with many positive results (in particular, we have nice inner models for measurability, for strong cardinals, for Woodin cardinals, and we have nice inner models of models of determinacy, that capture the large cardinals that provide us with the consistency of the determinacy statements).
To conclude, we understand (motivate/explain) large cardinals within the larger context of reflection principles, the simplest of which follow already from $\mathsf{ZFC}$. (So, we have a natural generating principle for them.) On the other hand, I know of no objections to large cardinals beyond "they are too large" or "they do not feel right", neither of which seems mathematical to me. The first also seems particularly artificial.
The only 'program' towards their inconsistency (that I am aware of) instead produced many interesting consequences for the partition calculus at the level of $0^\sharp$ (and is perhaps responsible for the early theory of $0^\sharp$ itself). As far as I understand, a similar attempt to disprove measurable cardinals resulted instead in the development of the covering lemma, which has since been one of the key tools to measure our understanding of particular large cardinals as part of the inner model program, see
Perhaps I should add that our intuitions about large cardinals do not come for free, but are the result of the programs mentioned above. I am in particular suspicious of a priori mistrust of large cardinals, since it tends to hide misunderstanding, or ignorance, of the actual mathematics involved in these programs.