The total number of ways to choose $3$ out of $8$ balls is $\binom83=56$
The number of ways to choose $2$ out of $3$ blue balls and $1$ out of $5$ red balls is $\binom32\cdot\binom51=15$
Hence the probability is $\dfrac{15}{56}$
Three approaches:
(1) This can be viewed as a hypergeometric distribution. The urn contiains
four red balls and five blue balls. Let $X$ be the number of red balls
among five balls drawn at random without replacement. To draw more blue balls
then red you need to evaluate $P(X \le 2).$ In R statistical software
this can be evaluated as follows:
phyper(2, 4, 5, 5)
## 0.6428571
(2) The equivalent answer can be obtained using a combinatorial argument:
$$\frac{{4 \choose 0}{5 \choose 5}+{4\choose 1}{5 \choose 4}+{4 \choose 2}{5 \choose 3}}{{9 \choose 5}} = \frac{1 + 20 + 60}{126} = 81/126 = 9/14 = 0.6428571$$
(3) An approximate value (to about 3 places) from simulating a million draws of five balls from
such an urn and counting the red balls can be obtained as follows:
set.seed(616)
m = 10^6; urn = c(1,1,1,1,1,2,2,2,2) # 1 = blue, 2 = red
r = replicate(m, sum(sample(urn, 5)==2)) # sample 5 balls without replacement
mean(r <= 2) # mean of logical vector is nr of TRUEs
## 0.642822
The histogram below shows the simulated hypergeometric distribution of the number of red balls drawn. The open red dots show exact hypergeometric probabilities.
At the scale of the graph, it is not easy to see any difference between the
simulated and exact values.
Note: You should be sure you understand and can explain the details of either method (1) or method (2) for your class. The simulation is probably not something you are expected to know.
Best Answer
You are picking 4 balls without replacement. The probability that you pick four red balls is $$P =\frac{6}{11} \cdot \frac{5}{10}\cdot \frac{4}{9}\cdot \frac{3}{8}\\P=\frac{360}{7920}\\P=\frac{1}{22}\\P \approx 0.045 \\P \approx 4.5\%$$