Using the Polya Enumeration Theorem we get for the two cases using the
cycle index of the symmetric group with empty boxes
$$Q_1 = [R^3 B^4] Z\left(S_3;
(1+R+R^2+R^3)\times(1+B+B^2+B^3+B^4)\right)$$
and without
$$Q_2 = [R^3 B^4] Z\left(S_3; -1 +
(1+R+R^2+R^3)\times(1+B+B^2+B^3+B^4)\right).$$
Now the cycle index is
$$Z(S_3) = 1/6\,{a_{{1}}}^{3}+1/2\,a_{{2}}a_{{1}}+1/3\,a_{{3}}.$$
Doing the substitution we find
$$\bbox[5px,border:2px solid #00A000]{
Q_1 = 28 \quad\text{and}\quad Q_2 = 18.}$$
If we want to do these by hand, here is an example. We use the
alternate form
$$[R^3 B^4] Z\left(S_3;
\frac{1}{1-R}\frac{1}{1-B}\right).$$
We get from the first term of the cycle index
$$[R^3 B^4] \frac{1}{6}
\frac{1}{(1-R)^3}\frac{1}{(1-B)^3}
= \frac{1}{6} {3+2\choose 2} {4+2\choose 2} = 25.$$
We get from the second term
$$[R^3 B^4] \frac{1}{2}
\frac{1}{1-R^2}\frac{1}{1-B^2}
\frac{1}{1-R}\frac{1}{1-B}
= \frac{1}{2} (1+1)\times (1+1+1) = 3.$$
Here we have e.g. for the coefficient on $B^4$ the possibilities
$(B^2)^2 (B^1)^0,$ $(B^2)^1 (B^1)^2$ and $(B^2)^0 (B^1)^4.$
At last we get from the third term
$$[R^3 B^4] \frac{1}{3}
\frac{1}{1-R^3}\frac{1}{1-B^3} = 0.$$
Add these to obtain $25+3=28.$
I think Ross Milikan's answer is incomplete. It doesn't seem to cover the (8,4) case you mention. Call f(N,M) the number of guesses you seek.After a bit of trial I find that I can break it down into four cases:
$$ \begin{array}{cc} (i) & \quad f(N,2) = \binom{N}{2}\\
(ii) & \quad M > N - \lfloor N/2 \rfloor \implies f(N,M) = N - M + 1\\
(iii) & N \textrm{ odd, } M = (N + 1)/2 \implies f(N,M) = 2 + (N - 1)/2 \\
(iv) & Otherwise\end{array} $$
In case $(iv)$ we have to do some recursion. You first set $f(N,M) = \binom{N}{2} - \binom{M}{2} + 1$, but then you check if you can make this guess any better as in your example from $f(8,4)$. For each $k$ for $k \in [3,\lfloor N/2 \rfloor ]$ Check if $$\binom{k}{2} + f(N-k,M-1) < f(N,M)$$ If so update $f(N,M)$ with this new value. Basically you can always take $k$ and check every pair. If these fail, then you know you had at most one red ball amongst them.
Best Answer
A: Choose $5+8=13$ boxes out of $20$ and multiply that by the number of ways of putting one red ball into $5$ of the chosen $13$ boxes. The rest of the chosen boxes get one blue ball each. Therefore the answer is $\binom{20}{13}\binom{13}{5}$.
B: Choose $5$ boxes out of the $20$ and put one red ball in each of them. Then choose $8$ boxes out of the $20$ and put one blue ball in each of them. The two ways of choosing the boxes are independent of each other, so we just multiply the obtained numbers to get the total number of ways to distribute the balls. Therefore the answer is $\binom{20}{5}\binom{20}{8}$.