what you are after if the coefficient of $x^{10}$ in the expansion
$$\begin{align}[x^{10}](1-x)^{-5}(1-x^5)^5 &=[x^{10}] \left( 1 + {5 \choose 1}x + {6 \choose 2 }x^2 + \cdots \right)\left(1 - {5 \choose 1}x^5 + {5 \choose 2}x^{10} + \cdots\right)\\&={5 \choose 2} - {9 \choose 5} {5 \choose 1}+ {14 \choose 10}.
\end{align} $$
Well, the number of ways you can arrange $a,a,a,b,b$ amounts to the number of ways to choose the locations for the three $a$s, since the $b$s will just go in the other two spots. But choosing locations for the $a$s is simply choosing a $3$-element subset of the set of all $5$ locations, of which there are $C[5,3],$ as you noticed.
Let's number our spots--$\{1,2,3,4,5\}.$ Then if we went with the arrangement $aabab,$ this would correspond to the subset $\{1,2,4\},$ for example. On the other hand, the subset $\{2,3,5\}$ would correspond to the arrangement $baaba.$
As you already observed, this only works with two types of objects to permute, since the locations of the first type of object tell the whole story. Otherwise, we can analogously use multinomial coefficients. (Note that the article uses "permutations with repetition" differently than you do.)
Best Answer
Consider $$(1-x)^8=1-8 x+28 x^2-56 x^3+70 x^4-56 x^5+28 x^6-8 x^7+x^8.$$ The highest power in this polynomial is $x^8$. Thus the coefficient of $x^{12}$ is $0$.
Consider $${1\over (1-4x)^5}=\sum_{n=0}^\infty{n+5-1\choose n}4^nx^n.$$ Let $n=12$. The coefficient of $x^{12}$ is ${16\choose 12}4^{12}=30534533120$.
We know that $${1\over (1-x)^k}=\sum_{n=0}^\infty{n+k-1\choose n}x^n.$$ We can use this fact to calculate the coefficients of $x^n$ as we please.