Every once in a blue moon it actually matters that some mathematical entity which might a priori only be a class is in fact a set. For clarification, here are some examples of what I do not mean:
A) Some colleagues of mine once made the following disclaimer: 'The "set" of stable curves does not exist, but we leave this set theoretic difficulty to the reader.' These colleagues (names withheld to protect the innocent) are of course fully aware of the fact that strictly speaking, the class of all stable curves, or topological spaces, or groups, or any the other usual suspects customarily formalized in terms of structured sets, cannot itself be a set. While they recognize that the structure they study is transportable along arbitrary bijections between members of a proper class of equinumerous sets, they also recognize that in their setting this same transportability could justify a technically sufficient a priori restriction to some fixed but otherwise arbitrary underlying set: that is, the relevant large category has a small skeletal subcategory. (Exercise: precisely what makes this work in the example given?) In such cases, the set versus class pecadillo is an essentially victimless one, perhaps barring discussions of the admissibility of Choice, expecially Global Choice.
B) There are various contexts in which seemingly unavoidable size issues are managed through the device of Grothendieck Universes. Such a move beyond ZFC might be regarded as cheating, sweeping the issue under the carpet for all the right reasons. Allegations of this nature regarding the use of derived functor cohomology in number theory, as in the proof of Fermat's Last Theorem, can now be laid to rest, as Colin McLarty has nicely shown in "A finite order arithmetic foundation for cohomology" http://arxiv.org/abs/1102.1773.
C) Set theory itself is replete with situations where the set versus class distinction is of paramount importance. For just one example, my very limited understanding is that forcing over a proper class of conditions is not for the unwary. I'd be interested to hear some expert elucidation of that, but my question here is in a different spirit.
With these nonexamples out of the way, I have a very short list of examples that do meet my criteria.
1) Freyd's theorem on the nonconcretizability of the homotopy category in "Homotopy is not concrete" http://www.tac.mta.ca/tac/reprints/articles/6/tr6abs.html. By definition, a concretization of a category is a faithful functor to the category of sets. The homotopy category (of based topological spaces) admits no such functor.
The crux of the argument is that while any object of a concretizable category has only a set's worth of generalized normal subobjects, there are objects in the homotopy category – for example $S^2$ – which do not have this property (page 9). The original closing remark (page 6) mentions another nonconcretizability result, for the category of small categories and natural equivalence classes of functors. A purist might try to disqualify the latter as too `metamathematical', but the homotopy example seems unassailable.
2) A category in which all (co)limits exist is said to be (co)complete; a bicomplete category is one which is both complete and cocomplete. Freyd's General Adjoint Functor Theorem gives necessary and sufficient conditions for the existence of adjoints to a functor $\Phi:{\mathfrak A}\rightarrow{\mathfrak B}$ with $\mathfrak A$ (co)complete. Let us say that a functor which preserves all limits is continuous, and that one which preserves all colimits is cocontinuous. A bicontinuous functor is one which is both continuous and cocontinuous.
Let us say that $\Phi$ is locally bounded if for every $B\in {\rm Ob}\,{\mathfrak B}$ there exists a set $\Sigma$ such that for every $A\in{\rm Ob} \,{\mathfrak A}$ and $b\in{\rm Hom}_{\mathfrak B}(B,\Phi A)$ there exist $\hat{A}\in{\rm Ob}\,{\mathfrak A}$ and $\hat{b}\in{\rm Hom}_{\mathfrak B}(B,\Phi\hat{A})\cap\Sigma$ such that
$b=(\Phi \alpha)\hat{b}$ for some $\alpha\in{\rm Hom}$ $_{\mathfrak A}$ $(\hat{A},A)$, and that $\Phi$ is locally cobounded if for every $B\in {\rm Ob}\,{\mathfrak B}$ there exists a set $\Sigma$ such that for every $A\in{\rm Ob}\,{\mathfrak A}$ and $b\in{\rm Hom}_{\mathfrak B}(\Phi A,B)$ there exist $\hat{A}\in{\rm Ob}\,{\mathfrak A}$ and $\hat{b}\in{\rm Hom}_{\mathfrak B}(\Phi\hat{A},B)\cap\Sigma$ such that $b=\hat{b}(\Phi \alpha)$ for some $\alpha\in{\rm Hom}_{\mathfrak A}(A,\hat{A})$. In the literature these are known as the Solution Set Conditions.
Theorem.
Let $\Phi:{\mathfrak A}\rightarrow{\mathfrak B}$ be a functor, where $\mathfrak B$ is locally small.
$\star$ If $\mathfrak A$ is complete then $ \Phi$ admits a left adjoint if and only if $\Phi$ is continuous and locally bounded.
$\star$ If $\mathfrak A$ is cocomplete then $ \Phi$ admits a right adjoint if and only if $\Phi$ is cocontinuous and locally cobounded.
See pages 120-123 of MacLane's "Categories for the working mathematician".
The local (co)boundedness condition has actual content. For example:
a) The forgetful functor ${\bf CompleteBooleanAlgebra}\rightarrow{\bf Set}$ is continuous but admits no left adjoint.
b) Functors ${\bf Group}\rightarrow {\bf Set}$, continuous but admitting no left adjoint, may be obtained as follows: let $\Gamma_\alpha$ be a simple group of cardinality $\aleph_\alpha$ (e.g. the alternating group on a set of that cardinality, or the projective special linear group on a 2-dimensional vector space over a field of that cardinality) and take the product (suitably construed), over the proper class of all ordinals, of the functors ${\rm Hom}_{\bf Group}(\Gamma_\alpha,-)$.
c) Freyd proposed another interesting example (see page -15 of the Foreword to "Abelian categories" http://www.tac.mta.ca/tac/reprints/articles/3/tr3abs.html) of a locally small bicomplete category $\mathfrak S$ and a bicontinuous functor $\Phi:{\mathfrak S}\rightarrow {\bf Set}$ which admits neither adjoint: loosely speaking, the category of sets equipped with free group actions, and the evident underlying set functor.
Does anyone know of any other examples, especially fundamentally different examples?
Finally, one could focus critical attention on the very question posed. To what extent does the strength and flavor of the background set theory matter? Force of habit and comfort have me implicitly working in some material set theory such as ZF, perhaps a bit more if I want to take advantage of Choice, perhaps a bit less if I prefer to eschew Replacement. Indeed, I have actually checked that example b) may be formulated in the absence of Replacement: while the von Neumann ordinals are no longer available, the same trick already used to give a kosher workaround to the illegitimate product over all ordinals further shows that an appropriate system of local ordinals suffices for the task. I am also quite interested in hearing what proponents of structural set theory have to say.
Best Answer
In my experience in analysis, basically the only place where it is actually important to distinguish sets from proper classes arises when one wishes to invoke Zorn's lemma to locate a maximal object in some non-empty partially ordered set $X$ in which all chains are bounded (e.g. to create a maximal proper subspace, a maximal filter, a maximally defined bounded linear functional, etc.). Here it is crucial that $X$ is "small" enough to be an actual set (e.g. it is a collection of subsets of some space $V$ that is already known to be a set, or a collection of functions from $V$ to yet another set). For instance, one cannot use Zorn's lemma to construct a maximal set in the class of all sets, or a maximal group in the class of all groups, or a maximal vector space in the class of all vector spaces, despite the fact that in each of these classes, any chain has an upper bound (the direct limit). (Such maximal objects, if they existed, would soon lead to contradictions of the flavour of Russell's paradox or the Burali-Forti paradox; not coincidentally, one of the standard proofs of Zorn's lemma proceeds by contradiction, using the axiom of choice to embed all the ordinals into $X$, which can then be used to set up the Burali-Forti paradox.)
To put it another way: regardless of one's choice of foundations, it is clearly mathematically desirable to be able to easily locate maximal objects of various types; but it is obviously also desirable for the existence of such maximal objects to not lead (or mislead) one into paradoxes of Russell or Burali-Forti type. ZFC, with Zorn's lemma on one hand and the set/class distinction on the other, manages to achieve both of these objectives simultaneously. Presumably, many other choices of foundations (particularly those which are essentially equivalent to ZFC in a logical sense) can also achieve both objectives at once, but I usually don't see these points emphasised when such alternative foundations are presented in the literature.