The following is probably a bad question, but hopefully, it might have a very good answer.
In category theory there is a quite famous analogy between topoi and commutative rings, I was never convinced by this analogy, but the best way to see how far an analogy can be pushed is to challenge it. Clicking on the link that I provided above you can have an extensive presentation of the analogy, the general motto can be grasped by the following table.
Remark 6.1.1.3. $\space$ Let $\mathcal{X}$ be an $\infty$-category. The assumption that colimits in $\mathcal{X}$ are universal can be viewed as a kind of distributive law. We have the following table of vague analogies:
$$\begin{array}{ccc} && \text{Higher Category Theory} && \quad && \text{Algebra} && \\ \hline \\ & & \infty\text{-Category} & & & & \text{Set} \\ \\ & & \text{Presentable } \infty\text{-category} & & & & \text{Abelian group} \\ \\ & & \text{Colimits} & & & & \text{Sums} \\ \\ & & \text{Limits} & & & & \text{Products} \\ \\ & & \varinjlim(X_\alpha) \times_S T \simeq \varinjlim(X_\alpha \times_S T) & & & & (x + y)z = xz + yz \\ \\ & & \infty\text{-Topos} & & & & \text{Commutative ring} \end{array}$$
Definition 6.1.1.2 has a reformulation in the language of classifying functors ($\S$3.3.2):
That corresponds to Rem 6.1.1.3 in my version of HTT by Lurie.
Q. According to this analogy, what should be a field?
Maybe I should say why this might be a stupid question or even a stupid challenge for the analogy. In fact it might be the case that:
- The notion of field is interesting only in low dimension.
- The correct generalization of the notion of field looks very different in categories and trivializes for sets because of their intrinsic rigidity.
Best Answer
This is a long comment. I would prefer to say that (Grothendieck) topoi are "(some) affine schemes over $\text{Spec } \text{Set}$." Here is my preferred version of the table, sprinkle $\infty$s according to taste:
(Here all monoidal structures distribute over colimits, which I guess is equivalent to requiring that they be closed.)
This more general setup allows for "2-affine algebraic geometry"; for example, $\text{QCoh}(X)$ for $X$ a scheme (stack, derived stack, etc.) is now an example, and in some nice cases covered by Tannakian theorems this embedding of algebraic geometry into "2-ring theory" is even fully faithful.
We get closer to (Grothendieck) topoi by upgrading "symmetric monoidal" to "cartesian monoidal." If we upgrade "cartesian monoidal" to "has all finite limits" (and upgrading the functors to being left exact) we get almost all the way towards logoi (topoi and algebraic morphisms), which in this analogy are "(some) commutative $\text{Set}$-algebras." Topoi and geometric morphisms are the geometric objects corresponding to these commutative ring-like objects.
Example. Let $G$ be a discrete group and consider the logos $\text{Set}^G$ of $G$-sets. Algebraic morphisms from $\text{Set}^G$ to a logos $L$ correspond to $G$-torsors in $L$ (by Diaconescu's theorem), and accordingly "$\text{Spec } \text{Set}^G$" ($\text{Set}^G$ regarded as a topos) is a "2-affine" version of the stack $BG$.
It's not clear to me whether there's a compelling generalization of field here. One definition of a field is that it's a commutative ring with no nontrivial quotients (effective epimorphisms out); I don't know enough about topos theory to know if logoi have a useful notion of quotient or epimorphism.
A necessary condition might be "has at most one point," where here a point is a geometric morphism from / algebraic morphism to $\text{Set}$. This includes $\text{Set}$ (an avatar of $\mathbb{F}_1$?) and $\text{Set}^G$ for $G$ a group but excludes, for example, $\text{Sh}(X)$ for $X$ a topological space with at least two points.
I like Simon Henry's proposal that $\text{Set}$ is the only field. This would mean that $\text{Set}$ has no nontrivial "field extensions." It certainly seems to have no nontrivial "Galois extensions."