There’s one issue underlying a lot of the discrepancies between people’s answers, I think:
How are we defining “$f$ is a function $s \to V$”, where $s$ is a set and $V$ is a (possibly proper) class?
(hence also, how we define subsequent things like “a small-category-indexed diagram of sets”) There are at least two main options here:
$f$ is a class of pairs, such that…
$f$ is a set of pairs, such that…
At least in most traditional presentations, I think it’s defined as the latter, but some people here also seem to be using the former. The answer to this question depends on which we take.
If we take the “a function is just a class” definition, then as suggested in the original question, and as stated in François’ answer, we definitely have some big problems without replacement: Set is no longer complete and co-complete, etc. (nor are the various important categories we construct from it); we can’t easily form categories of presheaves; and so on. Under this approach, we certainly get crippling problems in the absence of replacement.
On the other hand, if we take the “a function must be a set” definition, we get some different problems (as pointed out in Carl Mummert’s comments), but it’s not so clear whether they’re big problems or not. We now can form limits of set-indexed families of sets; presheaf categories work how we’d hope; and so on. The problem now is that we can’t form all the set-indexed families we might expect: for instance, we if we’ve got some construction $F$ acting on a class (precisely: if $F$ is a function-class), we can’t generally form the set-indexed family $\langle F^n(X)\ |\ n \in \mathbb{N} \rangle$.
This is why we still can’t form something like $\bigcup_n \mathcal{P}^n(X)$, or $\aleph_\omega$. On the other hand, such examples don’t seem to come up (much, or at all?) outside set theory and logics themselves! Most mathematical constructions that do seem to be of this form — e.g. free monoids $F(X) = \sum_n X^n$, and so on — can in fact be done without replacement, one way or another.
Now… I seem to remember having been shown an example that was definitely “core maths” where replacement was needed; but I can’t now remember it! So if we take this approach, then we certainly still lose something; but now it’s less clear quite how much we really needed what we lost.
(This approach is very close to the question “What maths can be developed over an arbitrary elementary topos?”.)
This is a long comment. I would prefer to say that (Grothendieck) topoi are "(some) affine schemes over $\text{Spec } \text{Set}$." Here is my preferred version of the table, sprinkle $\infty$s according to taste:
- Categories and functors : sets
- Presentable categories and left adjoints : abelian groups
- Monoidal presentable categories and monoidal left adjoints : rings
- Symmetric monoidal presentable categories and symmetric monoidal left adjoints : commutative rings
(Here all monoidal structures distribute over colimits, which I guess is equivalent to requiring that they be closed.)
This more general setup allows for "2-affine algebraic geometry"; for example, $\text{QCoh}(X)$ for $X$ a scheme (stack, derived stack, etc.) is now an example, and in some nice cases covered by Tannakian theorems this embedding of algebraic geometry into "2-ring theory" is even fully faithful.
We get closer to (Grothendieck) topoi by upgrading "symmetric monoidal" to "cartesian monoidal." If we upgrade "cartesian monoidal" to "has all finite limits" (and upgrading the functors to being left exact) we get almost all the way towards logoi (topoi and algebraic morphisms), which in this analogy are "(some) commutative $\text{Set}$-algebras." Topoi and geometric morphisms are the geometric objects corresponding to these commutative ring-like objects.
Example. Let $G$ be a discrete group and consider the logos $\text{Set}^G$ of $G$-sets. Algebraic morphisms from $\text{Set}^G$ to a logos $L$ correspond to $G$-torsors in $L$ (by Diaconescu's theorem), and accordingly "$\text{Spec } \text{Set}^G$" ($\text{Set}^G$ regarded as a topos) is a "2-affine" version of the stack $BG$.
It's not clear to me whether there's a compelling generalization of field here. One definition of a field is that it's a commutative ring with no nontrivial quotients (effective epimorphisms out); I don't know enough about topos theory to know if logoi have a useful notion of quotient or epimorphism.
A necessary condition might be "has at most one point," where here a point is a geometric morphism from / algebraic morphism to $\text{Set}$. This includes $\text{Set}$ (an avatar of $\mathbb{F}_1$?) and $\text{Set}^G$ for $G$ a group but excludes, for example, $\text{Sh}(X)$ for $X$ a topological space with at least two points.
I like Simon Henry's proposal that $\text{Set}$ is the only field. This would mean that $\text{Set}$ has no nontrivial "field extensions." It certainly seems to have no nontrivial "Galois extensions."
Best Answer
Fields are the simple objects in $\text{CRing}$.
Edit: Some philosophical remarks. Elements having inverses is a property and not a structure, so in some sense it's not obviously a good idea to treat the inverse as extra structure. Talking about group objects instead of just monoid objects can really only be done in cartesian monoidal categories; in general you instead want to talk about monoid objects with some extra property. For example, Poisson-Lie groups are not group objects in the category of Poisson manifolds (which is not cartesian monoidal). Similarly, Hopf algebras are not group objects in the category of coalgebras (which is again not cartesian monoidal).
For commutative rings "$x$ is invertible" should be thought of as "the ideal generated by $x$ is the unit ideal," and of course this holds for all nonzero $x$ if and only if there are no nontrivial quotients. This suggests that if we want to generalize the definition of a field to other categories similar to $\text{CRing}$ then we should try to generalize this condition.
Example. For graded rings the natural generalization is "the homogeneous ideal generated by $x$ is the unit ideal." This is true precisely for graded rings such that every nonzero homogeneous element is invertible, such as the ring of Laurent polynomials over a field.