This is a great question and I think I may have responded to a similar one months ago on the companion site Math Overflow. I don't recall exactly what I said there,but to answer your question:
Firstly, although I firmly believe mathematics has to be learned actively and looking up answers should be something you try to avoid, there has to be a practical limit. Most of us have gotten a stubborn streak with a particular homework problem as a student where we literally waste days trying to solve it-we're not gonna let it "beat" us. Part of it is just stubbornness, but beneath it is a deeper fear that our inability to solve a problem with no help is the dreaded "wall" that shows we ain't as good as we think we are and it's the first step towards ending up mopping floors outside our more brilliant classmate's office at Princeton. This is a lie, of course-with the exception of the truly gifted, all mathematics students struggle with proofs and computations.
More importantly,since we live in a Real World where there are deadlines on assignments and time limits on exams, such thinking will be very self destructive if it's not controlled. Finding oneself when time runs out on an exam having spent all the time on a single problem and getting a grade of 7 out of 100 for such stupidity is not a good day.
Personally,I think all textbooks regardless of level should come with complete solutions manuals. I know,I get a lot of flack for that,but I think having access to the solutions is a very good thing for students to have because it allows them to set a limit as to how long they'll work on a problem by themselves without solving it. "Oh,but then they'll just look up the answers and get an A." That's a facile argument to me because even if their professor is irresponsible enough to grade them solely on work they can look up, sooner or later, they will be required to find answers to even more difficult problems without access to solutions.
A middle ground solution to the corundum is to have textbooks with good, detailed hints. In my experience, a good, well-worded hint is usually good enough for a hard working mathematics student to point him or her in the right direction,they usually don't need more then that to get unstuck. But I'm getting off topic here.
My point is that although certainly you should make every possible effort to try and work things out yourself, there comes a point where it becomes self defeating and you have to either look up the solution or ask for guidance. How long you're willing to work before considering asking for help is a decision you have to make for yourself, but after being a student for some time,it's not hard to work out a reasonable boundary to set for yourself on this.
I'd like to close by telling you how I study. I generally make up a large, detailed batch of study cards-2 kinds for mathematics; one containing theorems and sample exercises and the other definitions and/or examples.The definition and example cards are the critical ones for studying. This is what furnishes the basis for understanding mathematics. For example, to understand the Cayley group isomorphism theorem, you have to understand what it means to have a permutation on a group. You can then try and re-express the result in terms of other concepts. For example, you can think of the Cayley theorem as stating there is a fundamental group action of every group on itself. However you do it-absorbing the definitions and what they mean is absolutely critical. Test yourself numerous times to see if you've absorbed and understand them.
For the theorem cards, this is where things get creative. Study them with a pen and paper in hand.I generally write the statement of the theorem on one side and the proof on the other. Do not look at the proof-try and determine the proof yourself directly from the definition. Wrestle with it as long as you possibly can before turning it over. Then if you can't reproduce it-put it on the bottom of the deck and move on to the next one. Do this until you have 2 separate piles: the ones you can prove and the ones you can't. Then do it again until you can prove it. This works,trust me.
There is a mathematical physics book called "The Road to Reality" by Roger Penrose for which a forum has been created, including discussion and solutions for exercises. You can find it here to see if this is something like what you had in mind
I have no idea about the legality of such a thing, although the creators of the previously mentioned forum received permission from the author of the book to create it.
I think something like this would be good to have for the purposes of self-study, although the impression I get after talking to professors (i.e. anecdotal evidence) is that authors want to encourage students to struggle through the problems first without having an easy resource to fall back on. (I think the reasoning for that is looking at an answer without thinking hard on the problem will spoil the "mathematical growth" associated with the exercise, and having solutions would be too tempting for some students.)
Best Answer
Ask a more experienced person. IMHO that's really the only option, and one of the reasons for this is that it is very important for a proof to communicate a result and its justification to another person. If the proof is good enough to convince yourself, that's a start, but the real test is whether you can express it in such a way as to convince someone else.
And BTW... the same applies if the textbook does have a solutions manual. Your proof is inevitably going to be different from the one in the book, and it takes a lot of experience and mathematical understanding to decide whether the differences are important or not.