The history of set theory from Cantor to modern times is well documented. However, the origin of the idea of sets is not so clear. A few years ago, I taught a set theory course and I did some digging to find the earliest definition of sets. My notes are a little scattered but it appears that the one of the earliest definition that I found was due to Bolzano in Paradoxien des Unendlichen:
There are wholes which, although they contain the same parts $A$, $B$, $C$, $D$,…, nevertheless present themselves as different when seen from our point of view or conception (this kind of difference we call 'essential'), e.g. a complete and a broken glass viewed as a drinking vessel. […] A whole whose basic conception renders the arrangement of its parts a matter of indifference (and whose rearrangement therefore changes nothing essential from our point of view, if only that changes), I call a set.
(The original German text is here, §4; I don't remember where I got the translation.)
According to my notes, Bolzano wrote this in 1847. Since Boole's An Investigation of the Laws of Thought was published a just few years later in 1854, it seems that the idea of sets was already well known at that time.
What was the earliest definition of 'set' in the mathematical literature?
Historical queries of this type are hopelessly vague, so let me give some more specific criteria for what I am looking for. The object doesn't have to be called "set" but it must be
an independent container object where the arrangement of the parts doesn't matter.
- It should also be fairly general in what the set can contain. A general set of points in the plane is probably not enough in terms of generality but if the same concept is also used for collections of lines then we're talking.
- It shouldn't have implicit or explicit structure. Line segments, intervals, planes and such are too structured even if the arrangement of the parts technically doesn't matter.
- It should be an independent object intended to be used and manipulated for its own sake. For example, the first time a collection of points in general position was considered in the literature doesn't make the cut since there was no intent to manipulate the collection for its own sake.
- It should be a definition. Formal definitions as we see them today are a relatively new phenomenon but it should be fairly clear that this is the intent, such as when Bolzano says "I call a set" at the end of the quote above.
- It should be mathematical concept. The strict divisions we have today are very recent but it should be clear that the sets in question are intended for mathematical purposes. Paradoxien des Unendlichen is perhaps more of a philosophical treatise than a mathematical one, but it is clear that Bolzano is considering sets in a mathematical way.
That said, any input that doesn't quite meet all of these criteria is welcome since the ultimate goal is to understand how the modern idea of set came to be.
Best Answer
Euler in Lettres à une princesse d'Allemagne sur divers sujets de physique et de philosophie, 17-24 feb 1761, writes about objects he calls spaces (my emphasis):
etc., and illustrates this with what we would call ensemblist diagrams (fig. 39 to 89), famously reproduced on Swiss banknotes. The applications he gives here are to everyday logic, so perhaps less mathematical than intended by the question. (I don't know if he ever wrote again on the subject.)
Edit: Something I didn't know is that these diagrams were themselves anticipated by Leibniz, as can be seen in his undated manuscript De formæ logicæ comprobatione per linearum ductus ($\simeq$ "Testing logical forms through line drawings") published in "Opuscules et fragments inédits de Leibniz: Extraits des manuscrits de la Bibliothèque royale de Hanovre", Alcan, Paris (1903), pp. 292-321.
Further edit: This was mentioned at an earlier question, pointing indirectly to Margaret E. Baron, A Note on the Historical Development of Logic Diagrams: Leibniz, Euler and Venn (1969).