The answer in the notes is correct. There are $\binom{50}3$ different sets of $3$ blue balls and $\binom{50}2$ different sets of $2$ red balls. Each of the $\binom{50}3$ sets of $3$ blue balls may be paired with any of the $\binom{50}2$ sets of $2$ red balls to form a set of $3$ blue and $2$ red balls, and every set of $3$ blue and $2$ red balls is formed in that way. Thus, there are $\binom{50}3\binom{50}2$ sets of $3$ blue and $2$ red balls. Since there are $\binom{100}5$ different sets of $5$ balls, the probability of drawing a set of $3$ blue and $2$ red balls is
$$\frac{\binom{50}3\binom{50}2}{\binom{100}5}\;,$$
exactly as it says in the notes.
It is true that there are $2^5$ different $5$-term sequences of the colors red and blue, and that $\binom53$ of them have $3$ blue and $2$ red terms, but that’s not what we’re counting. To see what goes wrong here, imagine that the bag contains only $5$ balls of each color. Now it’s clear that there are $5!$ ways to draw the color sequence BBBBB: you must draw the $5$ blue balls in some order. To get the color sequence RBBBB, however, you must first draw one of the $5$ red balls, then one of the $5$ blue balls, then one of the $4$ remaining blue balls, then one of $3$ blue balls left after that, and finally one of the last $2$ blue balls; you can do this in $4\cdot5\cdot4\cdot3\cdot2=480$ different ways. The two color sequences BBBBB and RBBBB are therefore not equally likely; in fact, the latter is four times as likely as the former, and you’re four times as likely to get it when you draw at random.
Another problem with your solution is that the problem isn’t about order: the balls are drawn as a set of $5$ balls, all at once, not as a sequence of $5$ balls.
All of the boxes contain $N - 1$ balls. This is just a complicated conditional probability problem. Lets look at a single box with $r$ red balls and $g$ green balls. What would the probability be of getting green on the second? Well it depends on whether or not you draw a red or green first. If you draw a red first, then there are $\left.p(\text{second green } \right| \text{ first red}) = \frac{g}{r + g - 1}$. However, if you draw a green ball first then you have one less green to choose from giving: $\left.p(\text{second green } \right| \text{ first green}) = \frac{g - 1}{g + r - 1}$. So what are the chances of each condition happening? $p(\text{first red}) = \frac{r}{g + r}$ and $p(\text{first green}) = \frac{g}{r + g}$. Therefore we can finally write:
\begin{align}
p(\text{second green}) =& \left.p(\text{second green } \right| \text{ first red})p(\text{first red}) + \left.p(\text{second green } \right| \text{ first green})p(\text{first green})\\
=& \frac{r}{r + g}\frac{g}{r+g-1} + \frac{g}{r + g}\frac{g-1}{r+g-1} = \frac{g(r + g - 1)}{(r + g)(r + g - 1)} = \frac{g}{r + g}
\end{align}
Not surprising that drawing the second green has just as good of a chance of being green as the first pick.
Therefore for each of the $N$ boxes you need to compute $p(\text{second green})$ (which is just the probability of drawing a green on the first try). Now the condition is that we choose box $r$ which has $p(\text{second green}) = p(\text{first green}) = \frac{N - r}{N - 1}$. The probability of choosing box $r$ among $N$ boxes is just $\frac{1}{N}$ which gives:
$$
p(\text{second green}) = \sum_1^N \frac{1}{N}\frac{N - r}{N - 1} = \frac{1}{N(N - 1)}\sum_1^r (N - r)
$$
The first sum is very easy (you're just summing the same number, $N$, $N$ times) $\sum_1^N N = N\cdot N = N^2$. The second part is easy if you remember the sum of the first $n$ consecutive integers is $\sum_1^n i = \frac{n(n + 1)}{2}$. So this gives:
$$
p(\text{green}) = \frac{N^2 - \frac{N(N + 1)}{2}}{N(N - 1)} = \frac{2N^2 - N^2 - N}{2N(N - 1)} = \frac{N^2 - N}{2\left(N^2 - N\right)} = \frac{1}{2}
$$
For part $2$), we actually already computed that above: $\left.p(\text{second green }\right|\text{ first green}) = \frac{g - 1}{g + r - 1}$. But now you need to sum over the condition that it could be any of the $N$ boxes (edit: However, the last box, box $N$, has $0$ green balls (and thus seeing green first means it definitely wasn't this box. So we should only sum over the first $N - 1$ boxes and divide by $N - 1$, not $N$.):
\begin{align}
\left.p(\text{second green }\right|\text{ first green}) =& \sum_1^{N - 1}
\frac{1}{N - 1}\frac{N - r - 1}{N - 2} \\
=& \frac{N(N - 1) - (N - 1) - \frac{N(N - 1)}{2}}{N(N - 2)} \\
=& \frac{2N(N - 1) - 2(N - 1) - N(N - 1)}{2(N - 1)(N - 2)}\\
=& \frac{N(N - 1) - 2(N - 1))}{2(N - 1)(N - 2)} \\
=& \frac{(N - 1)(N - 2)}{2(N - 1)(N - 2)} \\
=& \frac{1}{2}
\end{align}
This is only valid for $N > 2$ (since if $N = 1$ there are no balls in each box and if $N = 2$ there is only one ball in each box). This result just confirms that drawing balls are independent events.
Best Answer
In my opinion, the key to this problem is to forgo elegance and count, focusing directly on the # of green balls selected.
My answer will be
$$ \frac{N\text{(umerator)}}{D\text{(enominator)}}.$$
Clearly,
$$D = 4^6.$$
I will compute various numerators $N_1, N_2, N_3, N_4$. Then
$$N = N_1 + N_2 + N_3 + N_4.$$
Assume that the colors are represented by G,R,O, and B (for black).
On each line below, the number of ways that a distribution can occur will be counted.
$\underline{\text{3 Greens} : N_1}$
GGGBBB : $~ \binom{6}{3}.$
GGGBBR : $~ \binom{6}{3} \times 3.$
GGGBBO : $~ \binom{6}{3} \times 3.$
GGGBRO : $~ \binom{6}{3} \times 3!.$
$$N_1 = \binom{6}{3} \times 13.$$
$\underline{\text{4 Greens} : N_2}$
GGGGBB : $~ \binom{6}{4}.$
GGGGBR : $~ \binom{6}{4} \times 2.$
GGGGBO : $~ \binom{6}{4} \times 2.$
GGGGRO : $~ \binom{6}{4} \times 2.$
$$N_2 = \binom{6}{4} \times 7.$$
$\underline{\text{5 Greens} : N_3}$
GGGGGB : $~ \binom{6}{5}.$
GGGGGR : $~ \binom{6}{5}.$
GGGGGO : $~ \binom{6}{5}.$
$$N_3 = \binom{6}{5} \times 3.$$
$\underline{\text{6 Greens} : N_4}$
GGGGGG : $~ \binom{6}{6}.$
$$N_4 = \binom{6}{6} \times 1.$$
final answer:
$$\frac{ \left[\binom{6}{3} \times 13\right] ~+~ \left[\binom{6}{4} \times 7\right] ~+~ \left[\binom{6}{5} \times 3\right] ~+~ \left[\binom{6}{6} \times 1\right] } {4^6}. $$