What you describe seems to me to be a normal mode of mathematical
progress, and I would urge you simply to carry on! Ride that train
as far as you can.
It often happens that someone's mathematical results can be
improved or generalized in various ways, and when this is
possible, it is mathematically desirable that the generalization
be undertaken well.
You may be worried that the value of this work is less than some
other totally original work. If the generalizations are routine,
then indeed that may be true. But from what you say, this doesn't
seem to be your case. Many generalizations are not routine and
such work is definitely worth doing.
Finally, let me caution you to guard yourself against a certain
mistake that sometimes undermines motivation for a young
researcher. Namely, it often happens in mathematical research that
we begin in a state of terrible confusion about a topic; as
research progresses, things only gradually become clarified. After
hard work, we finally begin to understand what is the actual
question we should be asking; and then, after fitful starts and
retreats, we gain some hard-won insight; until finally, after
laborious investigation, we have the answer.
But alas — it is at this point that the crippling illness
strikes. Namely, because the researcher now understands the
problem and its solution so well, he or she begins to lose sight
of the value of the very solution that was made. The mathematical advance begins to
seem trivial or obvious, perhaps without value. Having solved the problem so well,
the mathematician becomes a victim of his or her own success.
Because all is now so clear, it is harder to appreciate the value
of the achievement that was made.
Please guard against this disease! Do not denigrate your
achievement simply because it seems easy after you have made it.
In many mathematical realms, the actual achievement in research is
that certain issues and ideas become easy to understand. Please look
upon the ease of the answer at the end as part of the achievement itself, and
think back to the initial state of confusion at the beginning of
the work to realize the value of what you have done.
So please carry on and ride that railroad as far as it will take
you.
The answer is essentially the same as how much should the average mathematician know about combinatorics? Or group theory? Or algebraic topology? Or any broad area of mathematics... It's good to know some, it's always helpful to know more, but only really need the amount that is relevant to your work. Perhaps a small but significant difference with foundations is that there is a natural curiosity about it, just like I'm naturally curious about the history and geography of where I live, even though I only need minimal knowledge in day-to-day life.
One does need to know where to go when deeper foundational questions arise. So one should keep a logician colleague as a friend, just in case. This entails knowing enough about foundations and logic to engage in casual conversation and, when the need arises, to be able to formulate the right question and understand the answer you get. This is no different than any other area of mathematics.
Some mathematicians may have more than just a casual curiosity about foundations, even if they work in a completely different area. In that case, learn as much as your time and curiosity permits. This is great since, like other areas of mathematics, foundations needs to interact with other areas in order to advance.
So, what do you need to have a casual conversation with a logician? Adjust to personal taste:
- Some understanding of formal languages and the basic interplay between syntax and semantics.
- Some understanding of incompleteness and undecidability.
- Some understanding of the paradoxes that led to the current state of set-theoretic foundations.
- Some understanding that logic and foundations does interact with your discipline.
To address the additional questions regarding sets. Personally, I don't think it's right to say that the notion of set is defined by foundations. It's a perfectly fine mathematical concept though it (sometimes confusingly) has two distinct and equally important flavors.
The main evidence for this point of view is that the notion of set existed well before Cantor and their use was common. Here is one of my favorite early definitions due to Bolzano (Paradoxien des Unendlichen, 1847):
There are wholes which, although they contain the same parts A, B, C, D,..., nevertheless present themselves as different when seen from our point of view or conception (this kind of difference we call 'essential'), e.g. a complete and a broken glass viewed as a drinking vessel. [...] A whole whose basic conception renders the arrangement of its parts a matter of indifference (and whose rearrangement therefore changes nothing essential from our point of view, if only that changes), I call a set.
(See this MO question for additional early occurrences of sets of various shapes and forms.)
What Bolzano describes is the combinatorial flavor of sets: it's a basic container in which to put objects, a container that is so basic and plain that it has no structure of its own to distract us from the objects inside it. There is another flavor to sets in which they are used to classify objects, to put into a whole all objects of the same kind or share a common feature. This usage is also very common and also prior to foundational theories.
Mathematicians use both flavors of sets, often together. For a variety practical reasons, foundational theories tend to focus on one, and formalize standard (albeit awkward) ways to accommodate the other.
So-called "material" set theories (ZFC, NBG, MK) focus on the combinatorial flavor of sets. To accommodate classification, these theories allow for as many collections of objects as possible to be put together in a set (with little to no concern whether this is motivated, necessary, or even useful).
So-called "structural" set theories (ETCS, SEAR, many type theories) focus on the classification flavor of sets. To accommodate combinatorics, these theories include a lot of machinery to relate sets and identify similar objects across set boundaries (with little to no concern about the nature of elements within sets).
Both of these approaches are viable and they both have advantages over the other. However, it's plainly wrong to think that working mathematicians have to choose one over the other, or even worry about the fact that it's difficult to formalize both simultaneously. The fact is that the sets mathematicians use in their day-to-day work are just as suitable as containers as they are as classifiers.
Best Answer
As others have indicated, the only 100% effective method of preventing getting "scooped" or finding out that your result already exists in the literature is that of complete abstinence: i.e., not trying to do any research at all.
Obviously this method is far too draconian for most of us on this site. I want to support statements of Gowers and Nielsen: finding out that what you have just proven has already been proven by someone else is far from the worst thing in the world. (Finding out that what you've proven, or published, is false, is much much worse, for instance.) On the contrary, for a mathematician who is making her own way and working on problems of interest to her, if you are doing any good work at all it is inevitable that you will duplicate some past research. This can be very encouraging: when I was younger, I often lacked confidence that some things which were of interest to me were of sufficient interest to anyone else (all I knew at that point was what people near to me were doing).
I remember in particular that as a first year graduate student, I discovered that every profinite group is, up to isomorphism of topological groups, the automorphism group of some Galois extension. This seemed neat but I thought, "Well, if anyone really cared, I would have heard about it before." Wrong -- this result has been published several times; off the top of my head by Leptin and by Waterhouse, but I know this list is not complete -- and in some texts (just not the ones I knew about at the time) it appears with due respect and appreciation. When I found out that someone had written and published a paper containing exactly the same mathematics that I had done, it was very encouraging.