I think that this question is somewhat similar to:
"How does a botanist decide which plants to look at?"
Well... a botanist always looks at lots of plants.
In the early phases of his/her career, a botanist will spend a lot a time studying well-known plants. Later on, a botanist will have learned to quickly recognise the commonly occurring plants so as to be able to quickly ignore them. He/she will then be able to focus their energy on rarer plants, or plants whose morphology is particularly interesting.
Even later, a botanist might have the opportunity to participate in expeditions to various remote parts of the planet, in order to search for plants that are new to science.
Now:
"How does a mathematician decide which problems to think about?"
Well... a mathematician thinks about a lot of stuff.
In the early phases of his/her career, a mathematician will spend a lot a time studying well-known constructions. Later on, a mathematician will have learned to recognise those problems that can be treated efficiently with well-known tools from those which are likely to be too hard. With the help of an adviser, he/she will then be able to focus their energy on problems that are at the right level of difficulty. Here, the meaning of "the right level of difficulty" always depends a lot on the expertise and background.
Experienced mathematicians have had the possibility to accumulate, throughout their interactions with colleagues, and by their own personal attempts at solving problems, a little collection of problems that are specialized enough so that no-one has really thought about them yet, and not too difficult for a graduate student. That's how graduate students often get their problems to work on.
The mathematicians who are no-longer graduate students have to constantly try to solve new problems. Their ability to find interesting problems (and to solve them) defines how good they are at their job.
Best Answer
Edit: The original answer below refers to Nelson's attempt from 2011. Upon a cursory look at the afterword by Sam Buss and Terence Tao to Nelson's paper placed in arxiv in 2015 (after his death), it seems he later attempted to address the error referred to in the original answer below; it would be interesting to know what the experts think on how successful his efforts were or potentially can be.
Original Answer: Edward Nelson's recent project on finding inconsistency of arithmetic (which was the subject of a MathOverflow Question) might be pertinent. The error, discovered by Terence Tao, seems to be the dependence of a constant on the underlying theory that Nelson did not account for.