Many students' first introduction to the difference between classes and sets is in category theory, where we learn that some categories (such as the category of all sets) are class-sized but not set-sized. After working with such structures, we discover that it is still worthwhile to sometimes treat them as if they were set-sized. But we can't always do so, without running into contradictions. A natural solution to this problem is the "Axiom of Grothendieck Universes". We then work not with the category of all sets, but the category of sets inside some given universe. This type of framework has been used by many notable mathematicians and seems very ubiquitous. For instance, it was an assumption in my universal algebra textbook. I would venture to say that most modern set theorists look at such an axiom as a very tame assumption.
My first question concerns the possible dangers of such an assumption, which are not often raised when first learning about this axiom. One obvious danger is that this axiom might lead to an inconsistency. In other words, ZFC might be consistent but ZFC+Universes might be inconsistent. I don't personally subscribe to this belief, but it is certainly a possibility (without further, even stronger, assumptions).
What really concerns me is the possibility of another danger: Could such a system proves false things about the natural numbers? In other words, even if we assume that ZFC+Universes is consistent, could it be the case that it proves false arithmetic statements?
The motivation for this question came from reading some of the work of Nik Weaver at this link, which argues for a conceptualist stance on mathematics. In particular, if we reject the axiom of power set, we are led to situations where all power sets of infinite sets are class-sized. Nik puts forth the idea that ZFC might prove false things about the natural numbers. Is this a real possibility? I suppose so, since it is possible that ZFC is inconsistent but the natural numbers aren't. But is it still possible even if ZFC is consistent? More generally:
Could treating the power set of the natural numbers as a set-sized object, rather than a class, force us to conclude false arithmetic statements (even if such a system is consistent)?
Even if the answer to this question is yes, I'm having a hard time seeing how we could recognize this fact, since according to the answers to this linked question it is difficult to state precisely what we mean by the natural numbers.
A second motivation for this question comes from what I've read about the multiverse view of set-theory. When creating a (transitive) model $M$ of ZFC, the set $P(\mathbb{N})$ can often be thought of as some "bigger" power set of the natural numbers intersected with the model. Moreover, via forcing, it seems that one can (always?) enlarge the power set of $\mathbb{N}$. This does seem to suggest that $P(\mathbb{N})$ is not completely captured in any model. Thus, in its entirety, perhaps $P(\mathbb{N})$ should be treated as a class-sized object.
Added: I believe that the current proof we have of Fermat's Last Theorem uses the existence of a(t least one) Grothendieck universe. However, my understanding is that this dependence can be completely removed due to the fact that Fermat's Last Theorem has small quantifier complexity. I imagine that proofs of statements with higher quantifier complexity that use Grothendieck universes, do not necessarily have a way of removing their dependence on said universes. How would we tell if such arithmetic statements are true of the natural numbers? [Some of this was incorrect, as pointed out by David Roberts and Timothy Chow.]
2nd addition: There are some theories that we believe prove false arithmetic statements. Assuming the natural numbers can consistently exist (which we do!), then both PA+Con(PA) and PA+$\neg$Con(PA) are consistent, but the second theory proves the false arithmetic sentence $\neg$Con(PA).
My question then might be rephrased as:
What principles lead us to believe that "Universes" is a safe assumption, whereas "$\neg$Con(PA)" is not safe, regarding what we believe is "true" arithmetic? (Next, repeat this question regarding the axiom of power set.) Is any theory that interprets PA "safe", as long as it is consistent with PA, and PA+Con(PA), and any such natural extension of these ideas?
Another way of putting this might be as follows:
Is the assumption Con(PA) a philosophical one, and not a mathematical one?
This ties into my previous question that I linked to, about describing the "real" natural numbers.
Best Answer
This answer repeats some of the material in other answers but I think it is not entirely redundant.
As you seem to have realized, the assertion that a theory is consistent is a much weaker statement than the assertion that the theory does not prove false arithmetical statements (or is "arithmetically sound", to use the standard terminology). So the answer to your question about whether it could be the case that ZFC + universes is consistent but not arithmetically sound is yes, but for perhaps a trivial reason: ZFC + universes could be consistent and yet ZFC itself could fail to be arithmetically sound. Probably what you really wanted to ask was, if ZFC is arithmetically sound, and ZFC + universes is consistent, does it follow that ZFC + universes is arithmetically sound? The answer is no. For example, it is conceivable that under these hypotheses, ZFC + universes could prove $\neg$Con(ZFC). (EDIT: This was a typo. I meant that ZFC+universes could prove $\neg$Con(ZFC+universes). Thanks to Noah Schweber for catching this—see the comments.)
Any feeling that universes are a "safe" assumption must come from a direct assessment of the universes axiom itself. The reasons are similar to the reasons that we have for believing that ZFC is arithmetically sound.
The ability to remove Grothendieck universes from the proof of Fermat's Last Theorem cannot be deduced from some simple abstract fact like "Fermat's Last Theorem has low quantifier complexity." No meta-theorem of this sort is known. Eliminability must come from careful examination of the nitty-gritty details of the proof.
Your question about whether Con(PA) is philosophical or mathematical needs clarification. Offhand, it sounds like a false dichotomy to me, predicated on dubious assumptions about the meaning of the words "philosophical" and "mathematical." Can you define precisely what you mean by a "mathematical" assumption? Are you implicitly assuming that "mathematical" assumptions are rock-solid while "philosophical" ones are not? If so, that in itself is a dubious assumption IMO.