There are $150$ balls altogether, and you’re choosing $20$ of them; a set with $150$ members has $\binom{150}{20}$ $20$-element subsets, so there are $\binom{150}{20}$ sets of $20$ balls that you could draw, all of them equally likely to be drawn. That accounts for the denominator of the fraction: it’s the number of equally likely possible outcomes.
Similarly, there are $\binom{40}{10}$ different sets of $10$ white balls, $\binom{50}4$ different sets of $4$ red balls and $\binom{60}6$ different sets of $60$ black balls. There are $$\binom{40}{10}\binom{50}4\binom{60}6$$ ways to combine one of the $\binom{40}{10}$ possible sets of $10$ white balls, one of the $\binom{50}4$ possible sets of $4$ red balls, and one of the $\binom{60}6$ possible sets of $6$ black balls, so there are $$\binom{40}{10}\binom{50}4\binom{60}6$$ successful outcomes, where successful means having $10$ white balls, $4$ red balls, and $6$ black balls. As usual, the probability of a successful outcome is the ratio of successful outcomes to equally likely possible outcomes, or
$$\frac{\binom{40}{10}\binom{50}4\binom{60}6}{\binom{150}{20}}\;.$$
Nothing in the analysis changes in any way if we paint numbers from $1$ through $150$ on the balls to give them unique identities: we still have to count the number of sets of $10$ white balls, the number of sets of $4$ red balls, the number of sets of $6$ black balls, and the number of sets of $20$ balls of whatever colors, and we still have to perform the same calculations with these numbers.
Added: I’m afraid that the calculation in the edit makes little sense. To see this more clearly, let’s look at a simpler example. Suppose that there are $99$ white balls and $1$ red ball, and you draw $20$ balls at random without replacement. By your reasoning there are $\binom{21}1=21$ possible outcomes, ranging from $20$ white and no red balls to no white and $20$ red balls. This, however, is clearly not the case, since there is only $1$ red ball in the bag. Okay, suppose that you take this limitation into account: then by your approach there are exactly two possible outcomes: $20$ white balls, and $19$ white balls and $1$ red balls, so the denominator of your fraction will be $2$.
For the outcome of drawing $20$ white balls the numerator will be $\frac{20!}{20!}=1$, so you would conclude that the probability of getting $20$ white balls is $\frac12$. Is that reasonable? Notice that you’d get the same result if there were $999$ white balls and $1$ red ball, or $999999$ white balls and $1$ red ball. I think that it’s pretty clear that this cannot be right.
Worse, your numerator for the outcome of drawing $19$ white balls and the red ball will be $\frac{20!}{1!19!}=20$, and the ‘probability’ of this outcome will be $\frac{20}2=10$. That certainly can’t be right!
What you’re missing is that even if there is nothing to distinguish one white ball from another, there are still $99$ different white balls in the bag, and a $99$-element set has $\binom{99}{20}$ different $20$-element subsets. Every one of these $\binom{99}{20}$ sets is a different outcome, even if you have no way to tell which of them you actually got. Thus, in the simplified problem the probability of getting $20$ white marbles is actually
$$\frac{\binom{99}{20}}{\binom{100}{20}}=\frac{99!}{20!79!}\cdot\frac{20!80!}{100!}=\frac{80}{100}=\frac45\;,$$
while the probability of getting $19$ white balls and the red ball is simply the probability of getting the red ball, $$\frac{20}{100}=\frac15\;.$$
Actually, $\binom{8}{4}$ is the correct value for $|S|$ (the way you're evaluating the probability, at least), but your value for $|E|$ is a bit off.
To see why your answer for $|E|$ is wrong, note that you're overcounting some selections several times. In particular, you're overcounting the case when you select all four red balls; for example, labeling the red balls $r_1, r_2, r_3, r_4$ and the black balls $b_1, b_2, b_3, b_4$, you count the selection $\{r_1, r_2, r_3, r_4\}$ four times; once when you choose $\{r_1, r_2, r_3\}$ as your first three red balls and $r_4$ as the fourth ball, once when you choose $\{r_1, r_2, r_4\}$ as your first three red balls, and $r_3$ as the fourth ball, etc.
The correct value for $|E|$ should be $\binom{4}{3}\cdot 4 + \binom{4}{4} = 17$; that is, the number of ways to choose 3 red balls and one black ball, plus the number of ways to choose all 4 red balls.
On the other hand, $|S|$ is indeed $\binom{8}{4}$. Since we labeled the balls, it doesn't matter that the balls are colored; the number we want is just the number of ways to choose 4 elements from a set of 8, which is just $\binom{8}{4}$.
Best Answer
You have $3!$ ways of ordering three different balls you have already drawn (first ball has three places to choose from etc.): $$ \{ABC, ACB, CAB, CBA, BCA, BAC\} $$ You have $2^3$ different color orders you can make when you draw balls from a bag (first ball can be one of two different colours etc.): $$ \{RRR, RRB, RBR, RBB, BRR, BRB, BBR, BBB\} $$ Correctly identifying what method to use to count exactly what you want to count (and not something else) is difficult and more or less only comes with experience. Keep up the work, and hopefully it sticks sooner or later.