An example of an important solution to a little-known problem might be
Frank P. Ramsey's "On a problem of formal logic" in Proc. London Math.
Soc. 30 (1930) 264-286. The problem was in logic and not well-known even
to logicians, but Ramsey's solution was taken up by combinatorialists
(notably Erdős and Szekeres) and it grew into the important field now known as
Ramsey theory.
{Added later] An example of the contrary type is Hilbert's fifth problem.
This was a well known and difficult problem, worked on by eminent
mathematicians such as von Neumann and Pontryagin, and it took
more than 50 years to solve. Yet, by the time it was solved it
seemed to be no longer in the mainstream of Lie theory, and books
on Lie theory today make little mention of it.
PS. I agree that this question should be community wiki.
Dear Alex,
It seems to me that the general question in the background of your query on algebra really is the better one to focus on, in that we can forget about irrelevant details. That is, as you've mentioned, one could be asking the question about motivation and decision in any kind of mathematics, or maybe even life in general. In that form, I can't see much useful to write other than the usual cliches: there are safer investments and riskier ones; most people stick to the former generically with occasional dabbling in the latter, and so on. This, I think, is true regardless of your status. Of course, going back to the corny financial analogy that Peter has kindly referred to, just how risky an investment is depends on how much money you have in the bank. We each just make decisions in as informed a manner as we can.
Having said this, I rather like the following example: Kac-Moody algebras could be considered 'idle' generalizations of finite-dimensional simple Lie algebras. One considers the construction of simple Lie algebras by generators and relations starting from a Cartan matrix. When a positive definiteness condition is dropped from the matrix, one arrives at general Kac-Moody algebras. I'm far from knowledgeable on these things, but I have the impression that the initial definition by Kac and Moody in 1968 really was somewhat just for the sake of it. Perhaps indeed, the main (implicit) justification was that the usual Lie algebras were such successful creatures. Other contributors here can describe with far more fluency than I just how dramatically the situation changed afterwards, accelerating especially in the 80's, as a consequence of the interaction with conformal field theory and string theory. But many of the real experts here seem to be rather young and perhaps regard vertex operator algebras and the like as being just so much bread and butter. However, when I started graduate school in the 1980's, this story of Kac-Moody algebras was still something of a marvel.
There must be at least a few other cases involving a rise of comparable magnitude.
Meanwhile, I do hope some expert will comment on this. I fear somewhat that my knowledge of this story is a bit of the fairy-tale version.
Added: In case someone knowledgeable reads this, it would also be nice to get a comment about further generalizations of Kac-Moody algebras. My vague memory is that some naive generalizations have not done so well so far, although I'm not sure what they are. Even if one believes it to be the purview of masters, it's still interesting to ask if there is a pattern to the kind of generalization that ends up being fruitful. Interesting, but probably hopeless.
Maybe I will add one more personal comment, in case it sheds some darkness on the question. I switched between several supervisors while working towards my Ph.D. The longest I stayed was with Igor Frenkel, a well-known expert on many structures of the Kac-Moody type. I received several personal tutorials on vertex operator algebras, where Frenkel expressed his strong belief that these were really fundamental structures, 'certainly more so than, say, Jordan algebras.' I stubbornly refused to share his faith, foolishly, as it turns out (so far).
Added again:
In view of Andrew L.'s question I thought I'd add a few more clarifying remarks.
I explained in the comment below what I meant with the story about vertex operator algebras.
Meanwhile, I can't genuinely regret the decision not to work on them because I quite
like the mathematics I do now, at least in my own small way. So I think what I had in mind was just
the platitude that most decisions in mathematics,
like those of life in general, are mixed: you might gain
some things and lose others.
To return briefly to the original question, maybe I do have some practical
remarks to add. It's obvious stuff, but no one seems to have written it so far on this page.
Of course, I'm not in a position to give anyone advice, and your question didn't really ask for it,
so you should read this with the usual reservations. (I feel, however, that what I write is an
answer to the original question, in some way.)
If you have a strong feeling about a structure or an idea, of course
keep thinking about it. But it may take a long time for your ideas
to mature, so keep other things going as well, enough to build up
a decent publication list. The part of work that belongs
to quotidian maintenance is part of the trade,
and probably a helpful routine for most people. If you go about it sensibly, it's really
not that hard either. As for the truly original
idea, I suspect it will be of interest to many people at some point, if
you keep at it long enough. Maybe the real difference between
starting mathematicians and established ones is the length of time
they can afford to invest in a strange idea before feeling
like they're running out of money. But by keeping a suitably interesting
business going on the side, even a young person can afford
to dream. Again, I suppose all this is obvious to you and many other people.
But it still is easy to forget in the helter-skelter of life.
By the way, I object a bit to how several people have described this question
of community interest as a two-state affair. Obviously, there are many different
degrees of interest, even in the work of very famous people.
Best Answer
(1) It depends a lot on the field. In fields that rely on specialized techniques discovered relatively recently or known only to a few, or fields where the questions involve recently-introduced objects, it's much easier to keep abreast of current research.
On the other hand, in fields with elementary questions that could have been studied a hundred years ago, sometimes even senior mathematicians discover that their work was studied a hundred years ago.
Of course, working in a trendy field carries its own risk, that someone else could be working on the same thing at the same time, but not much can be done about that.
(2) If you're working in a specialized field, as other have said, the best thing is to ask your advisor. If you have an advisor in a specialized field and have ideas in a different field, the best thing would be to ask someone in that field. As a grad student you probably want to start with fellow grad students, but a senior mathematician would probably asks someone on their own level.
If you have an idea that is more elementary, you should still ask your advisor, but there are certain mathematicians who know a lot of elementary and classical mathematics you could potentially ask.
(3) With regards to literature review, one trick that helps a bit when keyword searches fail is to use citations. If your idea generalizes work of Paper X, or answers a question from Paper X, or uses in a fundamental way the results of Paper X, anyone else who had the same idea would likely cite Paper X. You can produce a list of papers citing Paper X on both Google and MathSciNet.
(4) As a starting graduate student, even if your idea is completely new and original, it is likely that the greatest value it provides to you will be as practice for your future work. (I mean if you're good enough to do groundbreaking work right off the bat, you will probably do even more groundbreaking work once you get some experience under your belt.)
So don't feel bad at all if you find out something was already well-known - the experience of formulating and solving your own problem makes you well-placed to do original research once you learn a bit more, as compared to someone who knows a lot but hasn't done this.