I will give an answer, but first I would like to clarify the question. It seems to me that most commenters have misinterpreted the question. The question is not how people managed to construct different examples of moduli spaces before they had the tool of the language of functors. The question is the following: A moduli space is supposed to be a space whose points are in bijection with the isomorphism classes of some type of object. But it's not just that; for, if we only needed to find a bijection, then the sets would merely need to have the same cardinality. But we need more: namely, we need the geometry of the space to somehow reflect the nature of the objects in a "natural way." Now, we may have an intuitive idea of what this means, and in many cases we might be able to recognize when some space is not just in bijection with a class of objects but actually has geometry that reflects those objects. But the question is: how can we precisely state what this means?
Nowadays, we have the language of functors and functor of points, and we look not just at a single set, but at isomorphism classes of the given type of object over arbitrary bases, given a functor. We then say that a space is a moduli space for those objects if it represents that functor (or represents it up to isomorphisms - the coarse vs fine distinction is not too relevant for this discussion); note that now, this determines not only the set of points of our space, but it actually (by Yoneda's lemma) determines the geometry of our space.
So the question is the following: before the notion of functors and functor of points, how did people rigorously define what it meant for the geometry of the moduli space to reflect the geometry nature of the set of isomorphism classes of objects to be parametrized. I should add that this is a question that I have been curious about myself for a long time.
Now, according to Newstead's text Introduction to Moduli Problems and Orbit Spaces,
The word "moduli" is due to Riemann, who showed in his celebrated paper of 1857 on abelian functions that an isomorphism class of Riemann surfaces of genus $p$... However, it is only very recently that one has been able to formulate moduli problems in precise terms and in some cases to obtain solutions to them.
This book (at least the edition I'm looking at) was written in 1977, which gives some perspective on this statement.
In the ensuing chapter, Newstead goes on to define a family of objects parametrized by a variety. The definition is quite simple: it is a morphism of varieties $X \to S$ such that the fiber of any point $s$ of $S$ (i.e. its pre-image in $X$) is an object of the type in question. Even if one does not have the language of functors, my guess is that this idea could motivate a more precise notion of what a moduli space is.
János Kollár's draft book on moduli spaces also gives some hints:
The classical literature never differentiates between the linear system as a set and the linear system as a projective space. There are, indeed, few reasons to distinguish them as long as we work over a fixed base field $k$. If, however, we pass to a field extension $K/k$, the advantages of viewing $|L|$ as a $k$-variety appear.
The first sentence suggests that there was not a precise definition of moduli space in classical literature. It also suggests a natural idea leading to the functor of points, i.e. that over a field k, we might want to look and objects parametrized over different field extensions of k.
In general, as you can see from the history mentioned in his text, people didn't necessarily have precise definitions of moduli spaces, but they did understand that the geometry (well, the parameters) of the moduli space should correspond to the coefficients of the defining equations of the objects in question to be parametrized.
Finally, in Dieudonné's Historical Development of Algebraic Geometry, the author states on p.837
the precise meaning of this result that Riemann surfaces of genus $g$ are parametrized by $3g−3$ complex parametrized was to remain until very recently among the least clarified concepts of the theory.
While this doesn't answer the question, it might be of interest to note that Dieudonné later notes in his section on Grothendieck's functor of points
in particular, one transfers in that way to the theory of schemes many classical constructions such as projective spaces..., and one is able to give a general meaning to the concept of "moduli" introduced by Riemann for curves
suggesting slightly that there was no general meaning before this point.
That's the best I can do for now. An expert in the history of algebraic geometry might have more to say, but the sources I have seem to point in the direction of saying that there was not a precise definition until much later
Best Answer
Here is an overview of Euler's entries to the the annual prize competition of the Paris Academy. He came first 12 out of 15 years in the period 1727-1772.
Ronald Calinger describes Euler's first attempt to win this competition: