The problem here is that options are being over counted. For example you could pick:
BBBBP
then
BBBBB
or
BBBBB
then
BBBBP
and they would count as different. I think you may have overcomplicated things. The number of ways to pick 10 balls if you can't go over 5 red is the total number of ways to pick 10 balls (which you know) minus the number of ways to pick 10 balls if atleast 6 are red (which you know how to calculate).
a) Correct
b) No. You have 12 choices on the first item. 12 on the second. 12 on the third, fourth and fifth. So it should be $12^5$.
c) Correct
d) This is a bit tricky, I shall explain this below.
We have $C$ for combination, $P$ for permutation, and the first time I was taught this one (order does not matter + repetition allowed), my teacher called it $H$ although I am not sure if it is standard notation.
Anyway, the formula in general is $H_r^{n}=C_r^{n+r-1}$.
The reasoning behind this is as follows (for simplicity we shall take $H_5^{12}$ as in this case as an example):
Let the $12$ possible choices be $a,b,c,\dots,l$.
We don't care about the order here, we only care about how many of each item you choose, and with each choice, we can assign with it a unique "binary code".
For example, if you choose $aabbc$, then we write $0010010111111111$ where the first two $0$'s represent the two $a$'s, after the $1$ we have two $0$'s for the two $b$'s, and so on.
As another example, $abjkl$ is represented by $0101111111101010$.
In this way, each possible choice can be represented by a string of five $0$'s and eleven $1$'s for a total of $16$ numbers. So to count all possible choices, we simply have to choose the five places to insert the $0$'s.
This is given by $C_5^{16}$, hence $H_5^{12}=C_5^{16}$
Best Answer
Let’s say that the six types are A, B, C, D, E, and F. You’re going to pick ten of them. I’ll start with the questions that allow repetition, meaning that you can pick more than one object of a given type.
Say you pick $3$ A’s, a B, $4$ D’s, and $2$ F’s. If they’re not ordered, the list $\langle 3,1,0,4,0,2\rangle$, giving the number of items of each type, tells you everything that there is to know about the set. Counting the number of unordered sets of $10$ objects is the same as counting the number of $6$-tuples $$\langle n_A,n_B,n_C,n_D,n_E,n_F\rangle$$ that could describe such a set, where $n_A$ is the number of objects of type A in the set, $n_B$ is the number of objects of type B in the set, and so on. It’s not hard to see that the numbers $n_A,\dots,n_F$ must be non-negative integers, and they must sum to $10$: we want the number of integer solutions to the system
$$\begin{align*} &n_A+n_B+n_C+n_D+n_E+n_F=10\\ &n_A,n_B,n_C,n_D,n_E,n_F\ge 0\;. \end{align*}$$
This is what is sometimes called a stars-and-bars problem; the linked article gives the formula, from which we find that there are
$$\binom{10+6-1}{6-1}=\binom{15}5=3003$$ solutions and therefore $3003$ ways of picking an unordered set of $10$ objects if repetition is allowed. The article also gives a pretty decent explanation of the formula, and you can find other explanation on this site if you search for stars and bars.
Now suppose that we care not only about how many objects of each type we have, but also about the order in which we picked them. If you’re doing a multiple-choice test that has $10$ questions, each with $6$ choices, clearly choosing the answers in the order AAABDDDDFF is going to give you a different score from choosing them in the order ADFDDBAFAD! The appropriate calculation is actually much simpler. There are $6$ ways to choose the type of the first object. No matter which choice you make, there are $6$ ways to choose the type of the second object, so there are $6\cdot6=36$ ways to choose the types of the first two objects, just as there are $6\cdot6=36$ possible outcomes when you roll a red die and a green die. The same reasoning shows that there are $6^3$ possible ways to choose the types of the first three objects, and in general $6^k$ ways to choose the types of the first $k$ objects, so there are $$6^{10}=60,466,176$$ ways to choose the types of all $10$ objects.
Now suppose that repetition is not allowed. That simply means that you can choose at most one object of each type. In particular, in this problem it means that there’s no way to choose $10$ objects without repetition, whether you care about the order of choice or not. That’s easy, but it isn’t going to help much with problems in which the number of objects chosen is no bigger than the number of types, so let’s modify the problem and ask for $4$-element sets without repetition.
If they are unordered sets, all we care about is which of the six types are represented. The binomial coefficient $$\binom64=\frac{6!}{4!(6-4)!}=\frac{6!}{4!2!}=\frac{6\cdot5}{2\cdot1}=15$$ gives the number of $4$-element subsets of a $6$-element set, and that’s exactly what we want. More generally, the number of $k$-element subsets of an $n$-element set is $$\binom{n}k=\frac{n!}{k!(n-k)!}=\frac{\overbrace{n(n-1)(n-2)\ldots(n-k+1)}^{k\text{ factors}}}{k!}\;.\tag{1}$$
Finally, the numerator of $(1)$ gives the answer to the final question: there are
$$n(n-1)(n-2)\ldots(n-k+1)=\binom{n}k\cdot k!$$
ordered ways to choose $k$ distinct elements from a set of $n$ elements, i.e., not just to choose which $k$ elements are to be in our set, but also the order in which we list them. In the example, $n=6$ and $k=4$, so there are
$$\binom64\cdot 4!=15\cdot24=360$$
such sets.