The answer in the notes is correct. There are $\binom{50}3$ different sets of $3$ blue balls and $\binom{50}2$ different sets of $2$ red balls. Each of the $\binom{50}3$ sets of $3$ blue balls may be paired with any of the $\binom{50}2$ sets of $2$ red balls to form a set of $3$ blue and $2$ red balls, and every set of $3$ blue and $2$ red balls is formed in that way. Thus, there are $\binom{50}3\binom{50}2$ sets of $3$ blue and $2$ red balls. Since there are $\binom{100}5$ different sets of $5$ balls, the probability of drawing a set of $3$ blue and $2$ red balls is
$$\frac{\binom{50}3\binom{50}2}{\binom{100}5}\;,$$
exactly as it says in the notes.
It is true that there are $2^5$ different $5$-term sequences of the colors red and blue, and that $\binom53$ of them have $3$ blue and $2$ red terms, but that’s not what we’re counting. To see what goes wrong here, imagine that the bag contains only $5$ balls of each color. Now it’s clear that there are $5!$ ways to draw the color sequence BBBBB: you must draw the $5$ blue balls in some order. To get the color sequence RBBBB, however, you must first draw one of the $5$ red balls, then one of the $5$ blue balls, then one of the $4$ remaining blue balls, then one of $3$ blue balls left after that, and finally one of the last $2$ blue balls; you can do this in $4\cdot5\cdot4\cdot3\cdot2=480$ different ways. The two color sequences BBBBB and RBBBB are therefore not equally likely; in fact, the latter is four times as likely as the former, and you’re four times as likely to get it when you draw at random.
Another problem with your solution is that the problem isn’t about order: the balls are drawn as a set of $5$ balls, all at once, not as a sequence of $5$ balls.
Let's look at it like this. My combinatorics professor in college always emphasized on the semantics of mathematical language. What does this mean? Well since both of your pulls are separate but at the same time. You can do Box 1 and Box 2. In mathematical sense 'and' is ='+'. Since you have three colors of balls with a probably for each box. You will want to combined each probability, or 1 or 2. In this sense "or" = '$\times$'. You combine the probabilities of each like so
$$
(\frac{1}{8}\times\frac{5}{18})+(\frac{1}{3}\times\frac{7}{18})+(\frac{13}{24}\times\frac{1}{3})
$$
the result of which will be
$$
\frac{149}{432} = .34491 = 35\%
$$
if someone could check my work, but I think this might be the right answer. If this answer is incorrect could someone please supplement me with a response that explains my error. Again my apologies if it is incorrect Probability is something I haven't touched in a couple years.
Best Answer
$\left(\!\binom nk\!\right)$ denotes the number of choosing $k$ out of $n$ things with repetition allowed, and order not important; it is given by $\binom{n+k-1}{k}$. This is called multichoose, and it's where the "$2-1$" comes from.
The Wikipedia article on Stars and Bars is also helpful to understand this concept, there is a Numberphile video on YouTube too.
Solution: There are red, blue and green balls. You can choose at most one red ball. Thus if you choose the red ball, then you have $9$ balls left to choose, basically which of them are blue, and which are green. This can be done in $\left(\!\binom29\!\right) = \binom{2+9-1}{9}=10$ ways.
On the other hand, if you do not choose the red ball, then you choose 10 balls, either blue or green. This can be done in $\left(\!\binom2{10}\!\right) = \binom{2+10-1}{10}=11$ ways.
Thus the answer is $10+11=21$ ways.