Basically, to develop "formally" a geomety you have two ways; call them analytic and synthetic respectively.
Analytic
This is our "good old" Analytic geometry :
a point in the space is a ordered triple of real numbers : $(x_1,x_2,x_3)$
a line is the totality of points $(x_1,x_2,x_3)$ such that $u_1x_1 + u_2x_2 + u_3x_3 = 0$,
where at least one $u_j (j = 1,2,3)$ is different from zero
and so on ...
But real numbers are definable in set theory; thus - in principle - you can translate into set-theoretic notation the equation of the line.
Synthetic
See Edwin Moise, Elementary Geometry from an Advanced Standpoint (3rd ed - 1990), page 43 :
space will be regarded as a set $S$; the points of space will be the elements of this set. We will also have given a collection of subsets of $S$, called lines, and another collection of subsets of $S$, called planes.
Thus the structure that we start with is a triplet : $<\mathcal S, \mathcal L, \Pi>$, where the elements of $\mathcal S, \mathcal L, \Pi$, and are called points, lines and planes, respectively.
Our postulates are going to be stated in terms of the sets $\mathcal S, \mathcal L$, and $\Pi$.
Here are the first two postulates :
I-0 : All lines and planes are sets of points.
I-1 : Given any two different points, there is exactly one line containing them [we can "trivially" express the fact that the point $Q$ is contained into the line $l$ with the formula : $Q \in \mathcal S \land l \in \mathcal L \rightarrow Q \in l$ ].
We write $\overline{PQ}$ for the unique line containing $P$ and $Q$.
We define the relation of betweenness between (sic !) three points $P, Q, R$.
Then [see pages 64-65] : if $R,Q$ are two points, the segment between $R$ and $Q$ is the set whose points are $R$ and $Q$, together with all points between $R$ and $Q$.
The ray $\overrightarrow {AB}$ is the set of all points $C$ of the line $\overline {AB}$ such that $A$ is not between $C$ and $B$. The point $A$ is called the end point of the ray $AB$.
An angle is the union of two rays which have the same end point, but do not lie on the same line. If the angle is the union of $\overrightarrow {AB}$ and $\overrightarrow {AC}$, then these rays are called the sides of the angle; the [common] end point $A$ is called the vertex.
Finally, you can "close the circle" between this two approaches.
Assuming that we have defined the set $\mathbb N$ of natural numbers inside set theory [but I prefer to say that we have defined a model of the natural number system], and then the set $\mathbb R$ of real numbers, we can use $\mathbb R^3$ and call it : (three-dimensional) space.
Comment
What have we gained so far ? I think nothing more and nothing less than what we already have with Descartes' discovery of analytic geometry : an "embedding" of the euclidean geometry into the "cartesian plane".
Of course, the "basic" set-theoretic language gives us a powerful tool for expressing also geometrical "facts" : we can write $P \in l$ for : "the point $P$ is contained into line $l$", we can write $l_1 \cap l_2 \ne \emptyset$ for "two lines intersect each other", ...
But I think that speaking of "foundation for most of modern mathematics" can be mesleading.
I'll take a swing at answering the question that I think you are trying to ask. I'll formulate it as follows:
If foundations are important as the name suggests and the so-called
"foundational crisis" suggests, why do so few mathematicians concern
themselves much with them nowadays. If foundations aren't important,
then why was there a "foundational crisis" and a significant effort to
create foundations?
tl;dr "Foundations" and ZFC were created to solve a fairly specific problem (founding real analysis), which they did. Now we don't worry about the problem, so many mathematicians don't have much reason to "faff about" with foundations.
The first thing to note is the obvious statement that mathematics has been done before, during, and after the establishment of ZFC as a foundational system. Just as clearly, very little mathematics prior to the establishment of ZFC has been deemed "incorrect" since its establishment. (Even the parts that arguably may have been have often been "revitalized" in modern treatments, sometimes utilizing other foundational approaches, e.g. "infinitesimals".)
So the first point is "doing math" doesn't require a foundational system as witnessed by the fact that math was being done for thousands of years before the advent of ZFC or anything like it. This is also witnessed by the fact that you can learn quite a bit of math today without concerning yourself much with the details of ZFC.
My understanding of the situation near the "foundational crisis", which may well be wrong - I'm no math historian - is there was a fairly specific group that wanted something like set theory: real analysts (as we'd call them nowadays). My reading of the situation is that it was the controversies and vagaries in real analysis that sparked mathematical (as opposed to philosophical) interest in foundations. Intuitions about "real numbers", "functions", "continuous functions" were not enough for the mathematicians of the day to converge on questions like what the Fourier transform of the constant function should be or whether it should even exist. This also raised the possibility that the notion of "real numbers" itself might be incoherent.
This led to the early work on defining the reals and defining a notion of function. (There were also philosophical motivations for this work that were likely partially independent, but I suspect that without the issues in real analysis mathematicians would have largely ignored such work.) Of course, from there Russell's paradox scuttled the still largely intuitive conception of naive set theory. This likely also scuttled the idea that we could rely on mathematical intuition alone and reinforced the possibility that, e.g., the real numbers really could be built on quicksand. They certainly were in naive set theories. Then we had 40 years of many proposed foundational systems, modifications to those systems, critiques of those systems, meta-logical analyses of those systems, and hands-on work using the systems. Presumably the majority of mathematicians of the time were at most spectators to this. They continued to plod on doing math the way they'd always done it likely without much concern if they were, say, a (non-analytic) number theorist.
I would say the main go/no-go issue for a set theory of that time would be whether it could found real analysis, i.e. construct the real numbers, construct a notion of continuous function, and prove widely accepted results like Heine-Borel. Jumping to the modern day, yes, it is the case that the mere existence of any acceptable foundation removes much of the urgency of the "foundational crisis". Most mathematicians during the "foundational crisis" didn't care about sets, they cared about real numbers and continuous functions. Given some other (non-set-theoretic) framework to assuage their concerns, they would have had little interest in set theory. Nowadays, students (perhaps unfortunately...) don't have these concerns in the first place, so they have little reason to devote much or any time to foundations. Set theory is usually taught in a naive way with some warnings. The language and tools from set theory are useful even without it being a foundational system, so it's not ignored entirely.
The variety of foundational systems, the fact most of them are also capable of serving as a foundation for real analysis and most other branches of math, and the fact that ZFC itself goes far beyond what is needed by most mathematicians means most of mathematics doesn't really depend on the specific details of the underlying foundational system. For example, while finding an inconsistency in ZFC would be big news, it's hard to imagine that it would also impact all other foundational systems that are capable of supporting e.g. real analysis. It is likely that most results would be unaffected and the ones that were affected would be relatively easily adapted and still "morally true". Maybe an extra assumption is added, say.
Another aspect of this is that there are many results in more solidly grounded fields like number theory that have proofs that use mathematical objects in less solidly grounded fields like real analysis. To the extent these results have "elementary" proofs within the more solidly grounded field, we have a web of justification for the validity of non-trivial aspects of the less solidly grounded field. This puts some limits on how "wrong" we could be in those fields before we'd have to be "wrong about everything".
Best Answer
As many of the commenters have said, nothing is "wrong" with ZFC at a practical level; it's just that other theories have certain advantages. I'm not sure what you're referring to by "the strong desire to find something other than ZFC to use" — for the most part I've only seen people pointing out advantages of particular other theories rather than complaining that ZFC is broken on its own.
The first two disadvantages you point out are, as you say, not of practical importance for using ZFC as a foundation for mathematics. But they are not worthless observations either. If nothing else, they are important philosophical observations that there are ways in which ZFC doesn't match the everyday practice of mathematicians, so it ought to be of philosophical interest that there are other foundations that do match it better.
There is also a pedagogical point to make: the working mathematician may have no trouble ignoring an axiom, but a student just learning mathematics may get the wrong impression about how sets are actually used in mathematics from a ZFC-oriented introduction. (Indeed, pedagogical considerations were what first let Lawvere to invent ETCS.)
And there is also a question of extensibility: as we try to interpret a foundational theory in "nonstandard" models (I put the word in quotes because it has an undesired negative connotation), it is significantly easier if we use a foundational theory that looks more like the models that we are interested in as they arise naturally — namely, as categories, not as cumulative hierarchies. One can, with some effort, build a cumulative hierarchy from a category, but it doesn't always capture exactly what one wants, and why force oneself to go through the pain when there are other perfectly good foundational theories that we could use instead of ZFC?
As for category theory, there are at least two reasons a category theorist might be dissatisfied with set theory. One is the awkward treatment of universes used to deal with "large categories", but the alternative foundations that you mentioned (ETCS, HoTT) don't really do anything to solve that problem. (There are other alternative foundations, such as those proposed by Feferman, which do attempt to address that question, but I don't know as much about them.)
Another issue with set theory for category theory is that any two objects in set theory can be equal, whereas in category theory we generally only want to consider objects up to isomorphism, or categories up to equivalence, and so on into higher dimensions. You might think this is just like the question of nonsense statements $3\in \pi$, since you can just ignore the notion of equality and use isomorphism. However, technically you then incur an obligation to prove that all theorems and constructions respect isomorphism (or equivalence). Nobody actually does this in everyday mathematics, it being regarded as obvious; but when you start formalizing mathematics in a computer, it becomes necessary and tedious. ETCS and HoTT do have some advantages here: in ETCS formulated appropriately, one can prove a metatheorem that everything transports across isomorphism; whereas in HoTT this transportability is part of the basic theory (the univalence axiom).
To be fair, I should note that ZFC itself has some advantages over other foundational theories. In particular, it is very well-adapted for what the mathematicians who call themselves "set theorists" use it for, namely the study of well-founded relations. Most of the results of modern set theory could be formulated and proven in alternative foundations, but they often become significantly more awkward.