The answer in the notes is correct. There are $\binom{50}3$ different sets of $3$ blue balls and $\binom{50}2$ different sets of $2$ red balls. Each of the $\binom{50}3$ sets of $3$ blue balls may be paired with any of the $\binom{50}2$ sets of $2$ red balls to form a set of $3$ blue and $2$ red balls, and every set of $3$ blue and $2$ red balls is formed in that way. Thus, there are $\binom{50}3\binom{50}2$ sets of $3$ blue and $2$ red balls. Since there are $\binom{100}5$ different sets of $5$ balls, the probability of drawing a set of $3$ blue and $2$ red balls is
$$\frac{\binom{50}3\binom{50}2}{\binom{100}5}\;,$$
exactly as it says in the notes.
It is true that there are $2^5$ different $5$-term sequences of the colors red and blue, and that $\binom53$ of them have $3$ blue and $2$ red terms, but that’s not what we’re counting. To see what goes wrong here, imagine that the bag contains only $5$ balls of each color. Now it’s clear that there are $5!$ ways to draw the color sequence BBBBB: you must draw the $5$ blue balls in some order. To get the color sequence RBBBB, however, you must first draw one of the $5$ red balls, then one of the $5$ blue balls, then one of the $4$ remaining blue balls, then one of $3$ blue balls left after that, and finally one of the last $2$ blue balls; you can do this in $4\cdot5\cdot4\cdot3\cdot2=480$ different ways. The two color sequences BBBBB and RBBBB are therefore not equally likely; in fact, the latter is four times as likely as the former, and you’re four times as likely to get it when you draw at random.
Another problem with your solution is that the problem isn’t about order: the balls are drawn as a set of $5$ balls, all at once, not as a sequence of $5$ balls.
Question 1: It is easiest to prove this by induction. If $r = 0$ (there are no red balls) and $b \geq 1$, then clearly the first ball will be blue. That is, the number of balls $N$ that we need to draw in order to draw a blue ball has expected value $E(N) = 1$. Note that we can write this as
$$
E(N) = \frac{b+1}{b+1} = 1
$$
in accordance with the desired formula. Now, consider any other value of $r$, and $b \geq 1$ still. Suppose that we already have shown that for $r-1$ red balls and $b$ blue balls, the expected number of balls until we draw a blue ball is $\frac{b+r}{b+1}$, in accordance with the desired formula. We now draw a ball from an urn with $r$ red balls and $b$ blue balls. With probability $\frac{b}{b+r}$, the first ball is blue, and $N = 1$. With probability $\frac{r}{b+r}$, the first ball is red, and we now have an urn with $r-1$ red balls and $b$ blue balls. By the premise, we already know what the expected number of additional balls that need to be drawn in order to produce a blue ball; it is $\frac{b+r}{b+1}$. Therefore, for the urn with $r$ red balls and $b$ blue balls, we have
$$
\begin{align}
E(N) & = \frac{b}{b+r} \cdot 1
+ \frac{r}{b+r} \cdot \left(1+\frac{b+r}{b+1}\right) \\
& = \frac{b}{b+r} + \frac{r}{b+r} + \frac{r}{b+1} \\
& = 1 + \frac{r}{b+1} = \frac{b+r+1}{b+1}
\end{align}
$$
and we are done.
Question 2: By symmetry, the expected number of remaining balls left when only balls of one color (either red or blue) remain is equal to the number of balls of the same color we draw at the beginning. (To see this, draw all the balls out from the urn, and lay them out on a table, in sequence. The sequence from left to right is exactly as probable as the sequence from right to left.)
With probability $\frac{r}{b+r}$, the first ball is red. We are then left with $r-1$ red balls and $b$ blue balls, and we want to know how many balls we draw before (not until) we draw the first blue ball. This is one less than the result from Question 1. To this we have to add the first red ball. So when the first ball is red, our answer is
$$
N_{red} = \frac{b+r}{b+1}-1+1 = \frac{b+r}{b+1}
$$
With probability $\frac{b}{b+r}$, the first ball is blue, and by symmetry, the number of consecutive blue balls at the start is
$$
N_{blue} = \frac{b+r}{r+1}
$$
Thus, the expected number of consecutive balls drawn of the same color (which is also the answer to the original Question 2) is
$$
E(N) = \frac{r}{b+r} \cdot N_{red} + \frac{b}{b+r} \cdot N_{blue}
= \frac{r}{b+1} + \frac{b}{r+1}
$$
Best Answer
What you calculated is the probability that the first $3$ balls drawn are red, so your answer is wrong.
It is with replacement so there are $4$ independent experiments that all have the same probability to succeed (i.e. the draw results in a red ball). The probability on success is $\frac35$.
Then you are dealing with binomial distribution with parameters $n=4$ and $p=\frac35$.
To be found is: $$P(X=3)=\binom43\left(\frac35\right)^3\left(\frac25\right)^1$$
Observe that $\binom43=4$ and is actually the number of possibilities shown in the answer of Robert.