Dear Rohan,
First let me say that I got a little jealous after reading your first paragraph! What you are about to do should be very rewarding and a lot of fun.
I think you are right to be concerned that many students will find Rudin's book [as an aside, I don't really like this "Baby X" stuff: it seems not so subtly discouraging to describe university-level texts in this way; you could say either Rudin's Principles or, if you must use an epithet, "Little Rudin", because it is indeed a smaller book than his other analysis texts] too terse in and of itself. Sufficiently strong students will consider it a rite of passage and adapt to it eventually, but the entire current generation of "off-Rudin" undergraduate analysis texts seems to be fairly convincing evidence that the average undergraduate needs somewhat more help. Which is not to say that Berkeley is populated by average undergraduates, but I think even very strong students, whether they realize it or not, could learn more efficiently if the text is supplemented. (It happens that this was the primary text used in the math class I took during my very first quarter at the University of Chicago. It wasn't completely impenetrable or anything like that -- much less so than some of the lectures later on in the course! -- but I think I would have benefited from some of the supplementation you describe. In any case I'm too old to give a really independent evaluation of that book now: I've had it for getting on 20 years and have read much of it backwards and forwards countless times.)
The good news is that the off-Rudin phenomenon is so widespread that there are almost infinitely many places to go for a source of more problems, examples and so on. You really can have your pick of the litter. But since you asked, here are two books I like a lot, one old and one rather new:
Gelbaum and Olmstead, Theorems and Counterexamples in Mathematics.
What it claims is what you get, and what you get is very valuable. Asking students for counterexamples is a great way to keep them alive and awake in such a course: it's so easy for a young student to get snowed under by the barrage of the theorems and not to appreciate that so many theorems in real analysis have somewhat complicated statements because the simpler statement you are hoping for at first is simply not true. Coming up with counterexamples really helps students participate in the development of the material: if you don't do any of this explicitly, the very best students will do some of it on their own, but for a lot of the students learning the theorems will amount to a lot of arduous memorization.
Körner, A Companion to Analysis....
This is a pretty fantastic book: a long, chatty text much of which fills in nooks and crannies and works very hard to get the student to appreciate why things are set up they way they are. For instance, by now a lot of instructors have realized the pedagogical need to provide more up-front motivation for the real numbers and the obviously important but initially mysterious least upper bound axiom. Körner's book carries this line of thought through more deftly and thoroughly than any other I have seen. He asks the question "What happens if we try to do calculus on the rational numbers?" and he comes back again and again to answer it. It's tempting to throw out an example of a continuous function on a closed interval which attains its maximum value only at an irrational point and just move on, but this leaves a lot of cognitive work to the students to really appreciate what's going on. Körner does much better than this. Moreover, Körner's book ends with the best list of analysis problems I have ever seen. There are literally hundreds of pages of problems, thoughtfully organized and appealingly presented. This is an invaluable resource for someone trying to flesh out an analysis course.
I see that I've now written at length and not addressed most of your questions, which concern the solution sets. That may be for the best: it's been a long time since I regularly wrote up solutions to problem sets. This takes me back to my undergraduate days as well, when they were handwritten and mimeographed: yes, that was a pretty strange thing to do even back in the 1990's. (I will advise you tex up the solutions rather than handwriting them, although even this is not as de rigueur as one might think: I have colleagues who think that handwritten solutions are more appealing. I think they're crazy, but oh well.) I didn't get any feedback on them and often wondered if they were actually being read. So I had better leave this for someone else to advise you.
Good luck!
If your goal is to become a research mathematician, then doing exercises is important. Of course, there will be the rare person who can skip exercises with no detriment to their development, but (and I speak from the experience of roughly twenty years of involvement in training for research mathematics) such people are genuinely rare.
The other kinds of exercises that you describe are also good, and you should do them too!
The point of doing set exercises is to practice using particular techniques, so that you can recognize how and when to use them when you are confronted with technical obstacles in your research.
In my own field, two books whose exercises I routinely recommend to my students are Hartshorne's Algebraic geometry text and Silverman's Elliptic curves text. The exercises at the end of Cassels and Frolich are also good.
Atiyah and MacDonald also is known for its exercises.
One possible approach (not recommended for everyone, though) is to postpone doing exercises if you find them too difficult (or too time-consuming, but this is usually equivalent to too difficult), but to return to them
later when you feel that you understand the subject better. However, if upon return, you still can't fairly easily solve standard exercises on a topic you think you know well, you probably don't know the topic as well as you think you do.
If your goal is not to become a research mathematician, then understanding probably has a different meaning and purpose, and your question will then possibly have a different answer, which I am not the right person to give.
Best Answer
Why study set theory?
We like to think that mathematics developed from the need of our ancients to count things. I have four sheep, you have sixteen camels, my tribe has ten dozens of men, you have six hundred wives... etc. etc. But if you look closely, counting how many things you have of a certain type, first required you have be able and collect them into one collection. The "collection of all sheep I have", or the "collection of men in my tribe", and so on.
Sets came to solve a similar problem. Sets are collections of mathematical objects which themselves are mathematical objects.
This, of course, doesn't mean that we should learn set theory just for that purpose alone. The applications of set theory are not immediate for finite collections, or rather sufficiently small collections. We don't need to think about pairs or sets with five elements as particular objects. Whatever we want to do with them we can pretty much do by hand.
Sets come into play when you want to talk about infinite sets. Infinite sets collect infinitely many objects into one collection. The set of natural numbers, the set of finite sets of sets of sets of natural numbers, the set of sets of sets of sets of sets of sets of irrational numbers, etc. Once you establish that mathematical objects can be collected into other mathematical objects you can start analyzing their structure.
But here comes the problem. Infinite sets defy our intuition, which comes from finite sets. The many paradoxes of infinity which include Galileo's paradox, Hilbert's Grand Hotel, and so on, are all paradoxes that come to portray the nature of infinity as counterintuitive to our physical intuition.
Studying set theory, even naively, is the technical spine of how to handle infinite sets. Since modern mathematics is concerned with many infinite sets, larger and smaller, it is a good idea to learn about infinite sets if one wishes to understand mathematical objects better.
And one can study, naively, a lot of set theory, especially under the tutelage of a good teacher that will actually teach axiomatic set theory in a naive guise. And this sort of learning can, and perhaps should, include discussions about the axiom of choice, about ordinals, and about cardinals. As Ittay said, and I'm agreeing ordinals and cardinals are two ways of counting, which extend beyond our intuitive understand that counting is done via the natural numbers, and allow us to count infinite objects.
If one couples these ideas with the basics of first-order logic, predicate calculus, and basic first-order logic, one understands how set theory can be used as a basis for modern mathematics. Which again, allows us to better see into some parts of mathematics.
Axiomatic set theory, on the other hand, is a mathematical field like any other. It has certain type of typical problems, and set theorists work in their typical or atypical ways to solve them, or at least understand them better. Axiomatic set theory does, however, handle the fine-grained problems that come from infinity better.
Why do I mean by that? A lot of the infinite sets in modern mathematics are countable or have size continuum. Rarely we run into larger sets (e.g. the set of all Lebesgue measurable sets is larger), but even then we rarely care about that. But now that we understand infinite sets better, we can ask questions like "Given an abelian group with such and such properties, is it necessarily free [abelian]?" usually we can prove these sort of theorems for countable objects, in this case countable groups, but not beyond that.
Sometimes we are interested in topology, which allows us to extend our control from countable objects to things that can be approximated "in a good way" with countable objects (like separable spaces). But even then we can ask questions which involved an arbitrary objects, and not necessarily one which has 'nice properties'.
It turns out that our lack of intuition for infinite sets is reflected in the lack of "naively provable structure" of infinite sets. We cannot even provably determine how many distinct cardinalities lie between the cardinality of $\Bbb N$ and $\Bbb R$. It might be none, or it might be one or two or many more. Here axiomatic set theory comes into play.
Axiomatic set theory deals with the additional axioms that we can require the set theoretic universe to have, and how they affect the structure of infinite sets. And this is the importance of set theory to mathematical research, as well. It deals with solving the existence or what sort of assumptions we need to prove, or disprove, the existence of certain objects.
These objects, while seemingly arbitrary, can have a great influence and strong effects on the structure of "mathematically interesting sets". For example, we know that every Borel set is Lebesgue measurable. But the continuous image of a Borel set need not be Borel. Is it Lebesgue measurable? It turns out that yes, but if we close the Borel sets under complements and continuous functions, will the resulting sets be Lebesgue measurable? Will they satisfy some sort of "continuum hypothesis"? Will they have the Baire property? And other questions, which are all quite natural, originated all sort of strange set theoretical objects and axioms which assert their existence.
And if you ask me, that is why we should learn set theory, and what its importance is. It allows us to better understand infinite objects, and the assumptions needed to better control their behavior.