This is a very open-ended question, which may or may not have a perfect answer, and for which I have a few ideas but nothing like a clear picture. However, I guess it won't hurt to ask to see if people think about such things at all and if they do, what their ideas are. I don't know whether to make it CW or not: on one hand, it is pure mathematics, so we are within the usual set of standards to judge what's right and what's wrong, on the other hand, it is certainly not "a question of the type MO was designed for". So, I'm hesitant to check the community wiki option myself but have absolutely nothing against someone else doing so.
I assume that the ancient Greeks had an idea of a complete normed space ($\mathbb R$ and $\mathbb R^2$ would be enough for our purposes for quite a while), a set, a linear transformation, and the center of mass. On the top of it, I assume they had as much common sense (probably more), as we have nowadays.
The task is the usual one for Archimedes: given a reasonable non-empty set $E$ in a complete linear space $V$, assign a point $C(E)$ to it that you can confidently call "the center of mass". For the purposes of this thread, let's consider bounded at most countable subsets in $V=\mathbb R$ first. If we can figure out what to do with this case to everyone's satisfaction, we can move to the next stage. It may be not a really illuminating model, but it has a few quite funny features already.
The axioms of the center of mass are just the common sense ones:
1) The center of mass is never outside the closed convex hull of the set.
2) If $A$, $B$ are disjoint, then $C(A\cup B)\in[C(A),C(B)]$.
3) If $T$ is an affine transformation, then $C(TE)=TC(E)$.
4) (this is a bit tough, so feel free to drop or to modify it if it helps) If $A,B$ are such that the sets $a+B$, $a\in A$ are disjoint, then $C(A+B)=C(A)+C(B)$
I don't know if we really need anything else (in particular, I'm not sure if the addition of the "obvious" definition of the center of mass of a finite set is needed, helpful, or hurtful), but feel free to play with this list in any reasonably way you want.
The questions are the usual ones:
A) Existence
B) Uniqueness
C) Way to find $C(E)$ given $E$.
Any ideas, constructions, counterexamples, references, etc. (not necessarily restricted to the model I described) are welcome :).
Edit: Thanks to everyone who responded! Let me clarify one thing. Yes, if we can create a meaningful notion of mass (say, if there exists a translation invariant Borel measure that is finite and positive on $E$), we can define the center of mass in the usual way and the only question will be if the definition is unambiguous. However, if I give you a symmetric set, you'll not hesitate to say that the center of symmetry is also the center of mass. Also, if I give you an infinite set and add one point to it, you'll probably (but not necessarily) agree that the center of mass won't feel this addition, the reason being that the set was infinitely times more massive than the point we added not as mush because we can measure the actual mass in some way but merely because infinitely many disjoint shifts of the point fit inside the set. In other words, quite often we can determine the relative size without being able to assign any meaning to the absolute size. This is one of the loopholes in the integration theory I'd like to exploit and see how far one can get with it. In a sense, that is a straight extension of the original Eudoxus line of thinking, which is why "ancient Greeks" entered the title of the question.
Best Answer
If I'm not mistaken (but I often am), the physicists already have a rather simple way of defining the center of mass. But I don't think you can do it with just sets. You have to associate a mass with each set. The critical axiom is simply the one we all know:
If $A$ and $B$ are disjoint sets with masses $m(A)$ and $m(B)$ and center of masses $c(A)$ and $c(B)$ respectively, then the mass of $C$ is $m(C) = m(A) + m(B)$ and the center of mass of the set $C = A \cup B$ is given by $$ c(C) = \frac{m(A)}{m(C)}c(A) + \frac{m(B)}{m(C)}c(B). $$
You do need one more axiom to get started somehow. I believe physicists like to start with point masses (where the definition of the center of mass is easy) and then view a body as a limit of point masses. That's more or less what Liviu has proposed. But it also suffices to say that the center of mass of a square or cube is its geometric center. Or, more generally, the center of mass of any set with sufficient symmetry is its center.
Of course, if you really want arbitrary shapes, then you do need a countable version of the first axiom. But I think that's all you need. Note that this approach allows for bodies with different and even non-constant mass densities.
ADDED (in response to fedja's edit): It's worth noting explicitly that my answer above requires no notion of volume or choice of measure (such as Lebesgue measure) on the ambient space. It works on any length space.
But I don't see any way reduce this to just geometry (and not physics) without a notion of volume. In essence, you do it by just assuming all objects have the same constant mass density, so the mass is essentially equal to volume.
More generally, there has to be a way to measure the relative size of two sets, in order to determine the center of mass of the union of the two sets. If I understand correctly, axiom (4) in the question is an attempt to set this up.
[PREVIOUS DISCUSSION REPLACED BY THE FOLLOWING]
But for me it seems simpler to define a notion of size first and define the center of mass. And, as alvarezpaiva points out in a comment to Liviu's answer, valuations provide the appropriate setting, especially if we restrict to convex polytopes, which are objects that I believe the Greeks understood pretty well. This also allows us to avoid any issues of having to work with infinite sums or unions.
Here, a valuation $f$ is a finitely additive function on the space of convex polytopes. In other words, given polytopes $A$ and $B$, $$f(A \cup B) + f(A \cap B) = f(A) + f(B).$$ The first observation is that the "critical axiom" stated above is equivalent to saying that $C \mapsto m(C)c(C)$ is a valuation. However, Monika Ludwig showed in her paper Moment vectors of polytopes that the only $R^n$-valued measurable valuation on convex polytopes that behaves appropriately under affine transformations is the volume of the polytope times the standard center of mass.
Ludwig also showed in her Advances article Valuations on polytopes containing the origin in their interiors that any real-valued measurable $SL(n)$-invariant valuation homogeneous of positive degree must be a constant times volume. So it is reasonable to assume $m$ is volume. This therefore implies that $c$ must be the standard center of mass.
Moreover, if you examine Ludwig's proofs, you will see that although they are quite nontrivial, the technology used was arguably within the grasp of the Greeks.