This is a historical issue really.
Originally set theory was developed by Cantor and the well ordering principle was somewhat assumed in the background (e.g. Cantor's proof of the Cantor-Bernstein theorem was a corollary from the fact that every two cardinalities are comparable).
In 1904 Zermelo formulated the axiom of choice and proved its equivalence to the well ordering principle. He later formulated more axioms which described our intuition about sets, therefore removing the "naivity" from the Cantorian set theory. He did not add the axiom of foundations, nor the schema of replacement. Those were the result of Skolem and Fraenkel which were popularized by von Neumann.
The axiom of choice remained controversial, the thought that the continuum can be well-ordered was mind boggling and caused many people feel uneasy about this axiom. Further results like the Banach-Tarski paradox did not help to accept this axiom either.
Prior to set theory most mathematics was somewhat constructive in the sense that things were finitely generated or approximated by finitary means (e.g. limits of sequences). It requires quite the leap of faith to go from things you can pretty much write down to things which you cannot describe but only prove their existence. In this sense the axiom of choice augments the way we do mathematics by allowing us to discuss objects which we cannot describe in full.
It was questionable, therefore, whether this axiom is even consistent with the rest of the axioms of set theory. Gödel proved this consistency in the late 1940's while Cohen proved the consistency of its negation in the 1960's (it is important to remark that if we allow non-set elements to exist then Fraenkel already proved these things in the 1930's).
Nowadays it is considered normal to assume the axiom of choice, but there are natural situations in which one would like to remove it or find himself in universes where the axiom of choice does not hold. This makes questions like "How much choice is needed here?" important for these contexts.
Some things to read:
- Why worry about the axiom of choice? (MathOverflow)
- Advantage of accepting the axiom of choice
- Axiom of choice - to use or not to use
- Foundation for analysis without axiom of choice?
The question is broad, so I'll just try to address one of the sub-questions. I'll change it a little bit to allow for the possibility of pluralism:
What is the meaning of finding a "foundation of mathematics”?
As an example, I'll try to explain why set theory is a foundation of mathematics, and what it means for it to be one. This isn't intended to exclude other possible foundations, although they tend to be subsumed by set theory in a sense discussed below.
Based on one of the OP's comments above, I'll treat the mention of computability in the question figuratively rather than literally. Roughly speaking, a Turing-complete system of computation is one that can simulate any Turing machine, and so by the Church–Turing thesis, can simulate any other (realistic) system of computation. Therefore a Turing-complete system of computation can be taken as a "foundation" for computation.
As far as this question is concerned, I think that under the appropriate analogy between computation and mathematical logic, the notion of simulation of one computation by another corresponds to the notion of interpretation of one theory in another. By an interpretation of a theory $T_1$ in a theory $T_2$ I mean an effective translation procedure which, given a statement $\varphi_1$ in the language of $T_1$, produces a corresponding statement $\varphi_2$ in the language of $T_2$ such that $\varphi_1$ is a theorem of $T_1$ if and only if $\varphi_2$ is a theorem of $T_2$. So a mathematician working in the theory $T_1$ is essentially (modulo this translation procedure) working in some "part" of the theory $T_2$.
In these terms, a foundation of mathematics could be reasonably described as a (consistent) mathematical theory that can interpret every other (consistent) mathematical theory. However, it follows from the incompleteness theorems that no such "universal" theory can exist. But remarkably, the set theory $\mathsf{ZFC}$ can interpret almost all of "ordinary" (non-set-theoretic) mathematics.
For example, if I'm working in analysis and I prove some theorem about continuous functions, then my theorem and its proof can in principle be translated into a theorem of $\mathsf{ZFC}$ and its proof in $\mathsf{ZFC}$. (Under this translation, a continuous function is a set of ordered pairs, which are themselves sets of sets, satisfying a certain set-theoretic property, etc.)
This means that set theory can serve as a foundation for most of mathematics. For set theory itself, no one theory (e.g. $\mathsf{ZFC}$) can serve as a universal foundation. But we can informally define a hierarchy of set theories ($\mathsf{ZFC}$, $\mathsf{ZFC} + {}$"there is an inaccessible cardinal", $\mathsf{ZFC} + {}$"there is a measurable cardinal", etc.) called the large cardinal hierarchy, strictly increasing in interpretability strength, which seems to be practically universal in the sense that every "natural" mathematical theory, set-theoretic or otherwise, can be interpreted in one of these theories. (The reason this doesn't violate the incompleteness theorem is that the large cardinal hierarchy is defined informally in an open-ended way, so we can't just take its union and get a universal theory.)
Your last point about the continuum hypothesis raises an important question: what if the various candidate set theories branch off in many different directions rather than lining up in a neat hierarchy? Well, it turns out so far that the natural theories we consider do line up in the interpretability hierarchy, even if they do not line up in the stronger sense of inclusion. For example, the methods of inner models and of forcing used to prove the independence of $\mathsf{CH}$ from $\mathsf{ZFC}$ also give interpretations of the two candidate extensions $\mathsf{ZFC} + \mathsf{CH}$ and $\mathsf{ZFC} + \neg \mathsf{CH}$ by one other (and by $\mathsf{ZFC}$ itself,) showing that they inhabit the same place in the interpretability hierarchy. If you believe $\mathsf{CH}$ and I believe $\neg\mathsf{CH}$ we can still interpret each others results, rather than dismissing them as meaningless.
Best Answer
I'll take a swing at answering the question that I think you are trying to ask. I'll formulate it as follows:
tl;dr "Foundations" and ZFC were created to solve a fairly specific problem (founding real analysis), which they did. Now we don't worry about the problem, so many mathematicians don't have much reason to "faff about" with foundations.
The first thing to note is the obvious statement that mathematics has been done before, during, and after the establishment of ZFC as a foundational system. Just as clearly, very little mathematics prior to the establishment of ZFC has been deemed "incorrect" since its establishment. (Even the parts that arguably may have been have often been "revitalized" in modern treatments, sometimes utilizing other foundational approaches, e.g. "infinitesimals".)
So the first point is "doing math" doesn't require a foundational system as witnessed by the fact that math was being done for thousands of years before the advent of ZFC or anything like it. This is also witnessed by the fact that you can learn quite a bit of math today without concerning yourself much with the details of ZFC.
My understanding of the situation near the "foundational crisis", which may well be wrong - I'm no math historian - is there was a fairly specific group that wanted something like set theory: real analysts (as we'd call them nowadays). My reading of the situation is that it was the controversies and vagaries in real analysis that sparked mathematical (as opposed to philosophical) interest in foundations. Intuitions about "real numbers", "functions", "continuous functions" were not enough for the mathematicians of the day to converge on questions like what the Fourier transform of the constant function should be or whether it should even exist. This also raised the possibility that the notion of "real numbers" itself might be incoherent.
This led to the early work on defining the reals and defining a notion of function. (There were also philosophical motivations for this work that were likely partially independent, but I suspect that without the issues in real analysis mathematicians would have largely ignored such work.) Of course, from there Russell's paradox scuttled the still largely intuitive conception of naive set theory. This likely also scuttled the idea that we could rely on mathematical intuition alone and reinforced the possibility that, e.g., the real numbers really could be built on quicksand. They certainly were in naive set theories. Then we had 40 years of many proposed foundational systems, modifications to those systems, critiques of those systems, meta-logical analyses of those systems, and hands-on work using the systems. Presumably the majority of mathematicians of the time were at most spectators to this. They continued to plod on doing math the way they'd always done it likely without much concern if they were, say, a (non-analytic) number theorist.
I would say the main go/no-go issue for a set theory of that time would be whether it could found real analysis, i.e. construct the real numbers, construct a notion of continuous function, and prove widely accepted results like Heine-Borel. Jumping to the modern day, yes, it is the case that the mere existence of any acceptable foundation removes much of the urgency of the "foundational crisis". Most mathematicians during the "foundational crisis" didn't care about sets, they cared about real numbers and continuous functions. Given some other (non-set-theoretic) framework to assuage their concerns, they would have had little interest in set theory. Nowadays, students (perhaps unfortunately...) don't have these concerns in the first place, so they have little reason to devote much or any time to foundations. Set theory is usually taught in a naive way with some warnings. The language and tools from set theory are useful even without it being a foundational system, so it's not ignored entirely.
The variety of foundational systems, the fact most of them are also capable of serving as a foundation for real analysis and most other branches of math, and the fact that ZFC itself goes far beyond what is needed by most mathematicians means most of mathematics doesn't really depend on the specific details of the underlying foundational system. For example, while finding an inconsistency in ZFC would be big news, it's hard to imagine that it would also impact all other foundational systems that are capable of supporting e.g. real analysis. It is likely that most results would be unaffected and the ones that were affected would be relatively easily adapted and still "morally true". Maybe an extra assumption is added, say.
Another aspect of this is that there are many results in more solidly grounded fields like number theory that have proofs that use mathematical objects in less solidly grounded fields like real analysis. To the extent these results have "elementary" proofs within the more solidly grounded field, we have a web of justification for the validity of non-trivial aspects of the less solidly grounded field. This puts some limits on how "wrong" we could be in those fields before we'd have to be "wrong about everything".