Say you write a paper that you feel makes a borderline case for publication in a prestigious journal (Annals, Acta, Inventiones, JAMS, etc). What are the advantages and disadvantages of submitting the paper to the prestigious journal over a less prestigious journal where your paper is very likely to be accepted? My novice take is that there is little downside since if the paper is rejected you can always then submit it to a less prestigious journal. Are there factors I'm not considering? Of course, in most cases, the paper is Arxiv'ed early on so one's claim to the result is never hurt by a lengthy publication process.
I'd be interested to also hear about the implications on job applications. How much does it hurt a job applicant to have a very good result Arxiv'ed and submitted versus accepted? (Presumably if the validity of the result was a make or break factor for a job the hiring group could take a look at the preprint or talk to an expert in the field. Maybe that is an unrealistic expectation).
Best Answer
I somewhat disagree with some of the earlier answers on the "jobs applications impact". First, I think it makes absolutely no difference whether you have "submitted to Annals" or "preprint" on your CV. Everyone knows the acceptance chances and will ignore this line. Second, it is really important that all your papers are available on the arXiv or your personal web page. Often, the hiring committee can't judge the applicant's work, and will ask a local expert at the department to take a look at the papers and give an impartial opinion. If the papers are not available, the committee is forced to trust the applicant on their existence, a bad situation all around. Finally, except for the really top journals, having a paper published vs. having it still in a preprint form is of minor difference - if the local experts and/or reference letters are all saying that these recent papers are really good, that's sufficient. From that point of view, you should basically ignore the job application considerations, and always do what's best for the paper.
P.S. If I may make a suggestion, I think it's much more important to choose the right people to write reference letters than worrying about minor points in your CV. Especially now, in the mathjobs era of mass applications, it is the letters that really help people stand out from the crowd. So my advice would be to start thinking well in advance who can you ask for the letters, and learn how to better communicate your results (to them and everyone else).