This is a question of interest to most mathematicians who are research active and not slowly but surely knocking off important problems in their field at the rate of one per paper. (I think I could have ended the previous sentence at the word "active" without much affecting the meaning!)
I think the answer is ultimately quite personal: you are free to set your own standards as to how much of your work to publish. I myself understand the psychology both ways: on the one hand, math is usually long, hard work and when you finish off something, you want to record that accomplishment and receive some kind of "credit" for it. On the other hand, we want to display the best of what we have done, not the entirety. This position is well understood in the artistic and literary world: e.g. some authors spend years on works that they deem not ready to be released. Sometimes they literally destroy or throw away their work, and when they don't, their executors are faced with difficult ethical issues. (This is roaming off-topic, but I highly recommend Milan Kundera's book-length essay Testaments Betrayed, especially the part where he details the history of how after Kafka's death, his close friend Max Brod disobeyed Kafka's instructions and published a large amount of work that Kafka had specifically requested be destroyed. If Brod had done what he was told to do, the greater part of Kafka's Oeuvres -- e.g. The Trial, The Castle, Amerika -- would simply not exist to us. What does Kundera think of Brod's decision? He condemns it in the strongest possible terms!)
Another consideration is that publication of work is an effort in and of itself, to the extent that I would not say that anyone has a duty to do so, even after releasing it in some preprint form, as on the arxiv. A substandard work can be especially hard to publish in a "reasonable" journal. I have a friend who wrote a short note outlining the beginnings of a possible approach to a famous conjecture. She has high standards as to which journals are "reasonable", and rather than compromise much on this she determinedly resubmitted her paper time after time. And it worked -- eventually it got published somewhere pretty good, but I think she had four rejections first. I myself would probably not have the fortitude to resubmit a paper time after time to journals of roughly similar quality.
As you say, though, one advantage of formal publication is that the paper gets formal refereeing. Of course, the quality of this varies among journals, editors, referees and fields, but speaking as a number theorist / arithmetic geometer, most of my papers have gotten quite close readings (and required some revisions), to the extent that I have gained significant confidence in my work by going through this process. I have one paper -- my best paper, in fact! -- which I have rather mysteriously been unable to publish. It is nevertheless one of my most widely cited works, including by me (I have had little trouble publishing other, lesser papers which build on it), and it is a minor but nagging worry that a lot of people are using this work which has never received a referee's imprimatur. I will try again some day, but like I said, the battle takes something out of you.
Finally, you ask about how it looks for your career, which is a perfectly reasonable question to ask. I think young mathematicians might get the wrong idea: informal mathematical culture spends a lot of time sniping at people who publish "too many papers", especially those which seem similar to each other or are of uneven quality. Some wag (Rota?) once said that every mathematician judges herself by her best paper and judges every other mathematician by dividing his worst paper by the total number of papers he has published. But of course this is silly: we say this at dinner and over drinks, for whatever reasons (I think sour grapes must be a large part of it), but I have heard much, much less of this kind of talk when it comes to hiring and promotion discussions. On the contrary, very good mathematicians who have too few papers often get in a bit of trouble. As long as you are not "self plagiarizing" -- i.e., publishing the same results over and over again without admission -- I say that keeping an eye on the Least Publishable Unit is reasonable. Note that most journals also like shorter papers and sometimes themselves recommend splitting of content.
So, in summary, please do what you want! In your case, I see that you have on the order of ten other papers, so one more short paper which is in content not up there with your best work (I am going entirely on your description; I don't know enough about your area to judge the quality and haven't tried) is probably not going to make a big difference in your career. But it's not going to hurt it either: don't worry about that. So if in your heart you want this work to be published, go for it. If you can live without it, try that for a while and see how you feel later.
1) Has this happened to anyone else? Is this a relatively common occurrence, or am I just sloppy?
It is not very common (the usual preventive techniques include showing the draft to a few experts/friends, putting it on ArXiV, letting it lie for a month or two and then rereading it, etc. before sending it to a "top" journal) but it happens now and then. What is common is severe difficulty with finding an error in one's own work. I would say that affects more than a half of mathematicians I know. The reason is that you read not what is written but rather what you believe should be there when you just finish typing the manuscript and start proofreading. The main trick of good proofreading is to turn yourself into a complete idiot, who doesn't see a single step ahead, has no idea of the overall structure of the argument, takes everything literally, and is not convinced of anything that is not clearly put in a modus ponens form. Needless to say, it is about as hard for a shrewd person to read like that as for a genuinely stupid one to read between (or over) the lines. And even if you know all that, you are still destined to submit or even publish papers with errors. Just a few months ago, I was informed about an error in one of mine published papers. It was just a remark and the statement was actually correct, but the proof wasn't. So, to have this kind of public shame now and then is almost inevitable whether you are an unknown postdoc, or Andrew Wiles, or something in between.
I'm not sure if Poincare published a single formally correct proof in his lifetime and people still are completely puzzled by some passages in Linnik's works, so you are in a good company.
2) The anonymous referee is probably someone distinguished in my field. Do they now have a bad impression of me? (This probably is not a question that can easily be answered . . . .)
It is actually easy to answer: most likely, for him you are nobody, your name is just a random combination of letters, and your "initial value" is zero. An erratic paper leaves it this way, so nothing is lost. We are all getting worthless papers to referee every month and I challenge everyone to recall the name of the author of some bad paper he rejected 6 months ago. The only scenario in which "someone distinguished" would bother to take a mental record of your name after looking at a single opus of yours is when he finds something interesting and unusual in your work. Then your value for him is currently positive, though, of course, not as high as it would be if you solved the problem. So, again, there is absolutely nothing to worry about.
3) If I manage to patch up this paper, is it reasonable to resubmit it to this journal, or have I burned my bridges there?
Of course, it is. What matters is not how many mistakes you made on the road and who saw them but whether you finally reached your destination and whether other people consider that destination worth reaching. The theorem and its correct proof lose nothing in value if somebody published or tried to publish 20 false proofs before that. That some of those false proofs might be proposed by a person with the same name and biometrical characteristics as those of the one who finally found a correct proof changes nothing in the grand scheme of events. So, if you manage to fix the error and make sure that the argument is, indeed, correct, I see absolutely nothing wrong with submitting it again because from a purely logical standpoint, it is a different paper. If you get it returned solely on the grounds that the previous version was incorrect and not based on the merit considerations (even correct and good papers get rejected sometimes for various reasons), it'll merely tell you that the jornal is not really "top" but just "snobbish", in which case I would avoid it altogether in the future (at least, until they change the editorial board).
Best Answer
I once wrote a paper with an undergraduate that I thought was very nice. After it was accepted for publication, we found a paper not only proving our results, but going a step further. We hadn't found it previously because, similar to your situation, they used different terminology. In our case, our proofs didn't add anything new, and our results were weaker, so we withdrew the paper before it was published. That same undergraduate went on to write more papers with me, and is currently a successful graduate student. I mention this to emphasize that not publishing is not the end of the world.
If you decide to publish, you should definitely cite the work you found, and mention to the reader that their proofs are different and came first. Something like the following should be fine: "After a preprint version of this paper was made public I was made aware of [Y], which appeared first and obtained Theorems ###. As the methods this paper are completely different, we believe that the proofs still merit publication."
It is not necessarily your job to explain those differences, unless you think it would help the reader. However, it is your job to convince (yourself, the readers, and) the referee that your paper is worthwhile. So you need to deeply understand the differences, and perhaps explicitly point out what your paper adds to the literature.
I imagine that the authors of Y mentioned their paper because they have priority, and would appreciate citations to their work. After you have cited their work, mentioning their priority, you owe no more obligations to them. (Although, if they contact you again, you can treat their communications with respect!) Don't try to mind-read their intentions. Focus on your own beliefs, whether you personally think your proofs would help readers understand the topic more deeply and give them interesting techniques. If you like, you can send Y an updated version of your paper, which clearly gives them priority, and kindly ask if they have any further comments.