If a mathematician specializes in a popular research area, then there are many job positions available, but at the same time, many competitors who are willing to get such job positions. For an esoteric research area, there are few competitors and job positions. There are very often pros and cons of such research areas.
What are some pros and cons of specializing in esoteric research areas that many people may not know?
Maybe it is a little hard to answer this question in full generality since circumstances vary. Hence, I especially want to listen to examples, personal experiences, and maybe urban legends.
Now, if you are interested, let me tell my personal circumstance to give some context to this question. I am a student from outside of North America who just graduated from my undergraduate institution. Since I decided to study abroad, I applied to several North American universities last December and was admitted to some of them. Now I am wavering between two universities. Denote those universities X and Y.
Among the specific research areas available at X (resp. Y), I am interested in two of them, say, A and B (resp. C and D).
I did some searching in those research areas, and I found out that there are many people who are researching C and D and some of them are in my country. However, there are only a few people interested in A and B and none of them are in my country. Based on my search, I think (with a little exaggeration) there are about 3 universities in the world where a graduate student can specialize in A. B is not as esoteric as A, but still, it seems there are not many people working on B. However, I think C and D are quite major research areas in my field of study.
[In this question, I used 'field of areas' as something in First-level areas of Mathematics Subject Classification and 'specific research areas' as something in Second or Third-level areas of it. ]
At first glance, I prefer X over Y, because I was very interested in A. Also, this is partially (maybe totally) because X is considered more `prestigious’ than Y. However, I’m a little nervous about specializing in esoteric areas such as A and B, because of the number of job positions and this kind of problem.
Anyway, I think it is not a bad idea to ask a question at MO and to listen to the pros and cons of specializing in esoteric fields to make a better decision. Any personal stories or examples will be really helpful. Thanks in advance.
Best Answer
I think this is an important question, and something that is not talked about often enough. Just like we don't explain enough to undergraduates that the choice of their major has profound consequences for their career opportunities, we don't tell young mathematics researchers enough about the consequences of their choice of research area. (What's the analogue of the well-known joke about the English major serving food in your local restaurant?)
Instead of answering your question, let me give a small analysis of research dynamics, drawn from my own (limited) experience. The below is not supposed to be exhaustive, but I'm hoping to show that there are underlying mechanisms that cause certain areas to be more popular than others.
I'm hoping someone else will post an answer that gets closer to the core of your question, but I hope that this is at least somewhat useful.
What makes interesting research?
This may seem a little random when you're young, but there are actually some underlying mechanisms that are somewhat understandable. (Although admittedly I still haven't figured out a full answer to this question.)
First of all, just like universities and journals, top mathematics research gains some prestige from age. This means that solutions to old conjectures are valuable, and a new theory is appreciated more when it says something about the mathematics that existed before.
But there is a caveat: some areas are considered 'easy' or 'belonging to the past'. There are almost certainly areas that legitimately produce interesting mathematics but are not fashionable because they are considered 'too old'.
Secondly, mathematicians like powerful ideas. If you have some new technique that seems applicable in many situations, this is valued more than an ad hoc argument. If you have a powerful machine that people can use with their eyes closed, that's going to be cited a lot.
Thirdly, we prefer clean theorems. If you have a theorem without too many technical assumptions, then it's much easier to explain and motivate, and much easier for other people to use.
Moreover, connections to other areas are important too: if you prove amazing theorems on a (mathematical) island, then that's not as interesting as when you prove something that relates to other work.
A lot of the above seems driven by the fact that the mathematicians who evaluate your work do so on the basis of their own interests. This seems reasonable given that everyone's own expertise is limited, and their judgement biased towards their own work.
What makes fashionable research?
Areas of mathematics fall in and out of fashion, in part related to the criteria above. For example, if there is a new idea that seems to have potential, this can drive a research group until they feel the idea has been fully exploited. Especially new connections between different areas can spur a lot of activity, because now you have two communities working out these ideas.
When this happens, you can suddenly see a lot of researchers thinking about very similar things, and this is both a blessing and a curse. Indeed, it makes it easier to explain your results and to find collaborators, but it also means you have to keep up with progress all the time and risk getting 'scooped'.
I'm not entirely sure what the mechanisms for topics falling out of fashion are. One thing that can happen is that the main problem gets solved; for example I have the impression that the classification of finite simple groups ended an era of high activity in the area. But this doesn't always happen, because there could be other questions; for example the techniques for proving Fermat's Last Theorem are still very much alive in the Langlands programme.
I imagine it's also possible for a research area to dry out without the main problem being solved, although I haven't been around long enough to know an example offhand.
Note that timing is key: what is fashionable now may not be fashionable in the future, and sometimes researchers forecast the demise of their field. While it's hard to predict the future, it's probably a good idea to listen (critically) when people tell you something like this.
Finally, there are also some political factor involved: if your area is well-represented in the editorial boards of top journals, then you're going to have an easier time publishing in those journals. As Henriksen points out in There are too many B.A.D. mathematicians, this does not always happen for the right reasons.
So what to do next?
Many research groups are driven by a handful of famous questions and a much larger collection of more technical questions. You could ask the research groups you're considering what the main goal in their work is, and then evaluate this by the criteria above to see how easy a time you would have selling your research.
Usually the advice is to talk to some graduate students in the group to hear about their experiences, but in your case it might also be worthwhile to try to approach some postdocs¹ or senior researchers in the area. Another useful metric to look at is job placement: if an advisor has supervised many graduate students who did well on the academic market (short AND long term), this is useful information.
Finally, I should remark that these things matter much more when you're young, because the mathematics job market is highly catered towards the researchers who are lucky enough to have early success. Once you get a tenure-track of tenured job, it becomes easier to switch areas again. This is especially true if you work in an area with connections to other areas, although there are certainly examples of famous researchers switching to completely unrelated fields.
¹Be aware that postdocs are in the most precarious stage of their career, so their answers will be a bit more cynical than those of (young and naive) grad students or (successful) tenure-track or tenured faculty. This can be useful for getting more 'real' advice, but be sure to recognise the context in which it is given.