There are $150$ balls altogether, and you’re choosing $20$ of them; a set with $150$ members has $\binom{150}{20}$ $20$-element subsets, so there are $\binom{150}{20}$ sets of $20$ balls that you could draw, all of them equally likely to be drawn. That accounts for the denominator of the fraction: it’s the number of equally likely possible outcomes.
Similarly, there are $\binom{40}{10}$ different sets of $10$ white balls, $\binom{50}4$ different sets of $4$ red balls and $\binom{60}6$ different sets of $60$ black balls. There are $$\binom{40}{10}\binom{50}4\binom{60}6$$ ways to combine one of the $\binom{40}{10}$ possible sets of $10$ white balls, one of the $\binom{50}4$ possible sets of $4$ red balls, and one of the $\binom{60}6$ possible sets of $6$ black balls, so there are $$\binom{40}{10}\binom{50}4\binom{60}6$$ successful outcomes, where successful means having $10$ white balls, $4$ red balls, and $6$ black balls. As usual, the probability of a successful outcome is the ratio of successful outcomes to equally likely possible outcomes, or
$$\frac{\binom{40}{10}\binom{50}4\binom{60}6}{\binom{150}{20}}\;.$$
Nothing in the analysis changes in any way if we paint numbers from $1$ through $150$ on the balls to give them unique identities: we still have to count the number of sets of $10$ white balls, the number of sets of $4$ red balls, the number of sets of $6$ black balls, and the number of sets of $20$ balls of whatever colors, and we still have to perform the same calculations with these numbers.
Added: I’m afraid that the calculation in the edit makes little sense. To see this more clearly, let’s look at a simpler example. Suppose that there are $99$ white balls and $1$ red ball, and you draw $20$ balls at random without replacement. By your reasoning there are $\binom{21}1=21$ possible outcomes, ranging from $20$ white and no red balls to no white and $20$ red balls. This, however, is clearly not the case, since there is only $1$ red ball in the bag. Okay, suppose that you take this limitation into account: then by your approach there are exactly two possible outcomes: $20$ white balls, and $19$ white balls and $1$ red balls, so the denominator of your fraction will be $2$.
For the outcome of drawing $20$ white balls the numerator will be $\frac{20!}{20!}=1$, so you would conclude that the probability of getting $20$ white balls is $\frac12$. Is that reasonable? Notice that you’d get the same result if there were $999$ white balls and $1$ red ball, or $999999$ white balls and $1$ red ball. I think that it’s pretty clear that this cannot be right.
Worse, your numerator for the outcome of drawing $19$ white balls and the red ball will be $\frac{20!}{1!19!}=20$, and the ‘probability’ of this outcome will be $\frac{20}2=10$. That certainly can’t be right!
What you’re missing is that even if there is nothing to distinguish one white ball from another, there are still $99$ different white balls in the bag, and a $99$-element set has $\binom{99}{20}$ different $20$-element subsets. Every one of these $\binom{99}{20}$ sets is a different outcome, even if you have no way to tell which of them you actually got. Thus, in the simplified problem the probability of getting $20$ white marbles is actually
$$\frac{\binom{99}{20}}{\binom{100}{20}}=\frac{99!}{20!79!}\cdot\frac{20!80!}{100!}=\frac{80}{100}=\frac45\;,$$
while the probability of getting $19$ white balls and the red ball is simply the probability of getting the red ball, $$\frac{20}{100}=\frac15\;.$$
Best Answer
Out of the six flags, there are $\binom{6}{3} = 20$ ways to choose which flags are going to be white. Now out of the remaining three flags, pick which one is going to be blue, and the remaining two will automatically be red.