To explain where your formula comes from and why it works, consider breaking it up into steps:
Step 1: choose where the reds go. There are $\binom{10}{2}$ ways of arranging the reds and not reds (ignoring the fact that the not reds are of multiple colors for the moment).
Step 2: of the spaces labeled for use by not-reds, choose which of those spaces will be occupied by blues: There are $\binom{10-2}{3}$ number of ways to do this.
Step 3: of the spaces labeled for use by not-reds and not-blues, choose which are occupied by greens: There are $\binom{10-2-3}{5}$ number of ways.
Thus, there are $\binom{10}{2}\cdot\binom{8}{3}\cdot\binom{5}{5} = \frac{10!~~~8!~~~5!}{2!8!3!5!5!0!} = \frac{10!}{2!3!5!}$ number of ways to accomplish this.
(remember that $0!=1$ by definition)
Not sure this is the best method, but the way I would actually solve such a set of questions would be the following:
Use generating function methods for the second question.
The explicit collection of submultisets of the six tiles is given by the terms of $(1+x)(1+y+y^2)(1+z+z^2+z^3)$ (the $x$ terms correspond to tile 1, $y$ terms to copies of tile 2, and $z$ terms to copies of tile 3), and the numbers of submultisets of a given size are given ignoring the labels of the tiles and just multiplying the polynomials
$$(1+x)(1+x+x^2)(1+x+x^2+x^3)=(1+2x+2x^2+x^3)(1+x+x^2+x^3)$$
$$= 1+(1+2)x+(1+2+2)x^2+(1+2+2+1)x^3+(2+2+1)x^4+(2+1)x^5+x^6$$
$$=1+3x+5x^2+6x^3+5x^4+3x^5+x^6.$$
Looking at the coefficient of $x^3$ in the product tells us that there are 6 submultisets.
Now the question of ordered triples drawn from the multiset is a little more difficult. I think I'd work out the explicit submultisets of size 3 first. I.e. solve question 2, then compute the number of ways to order each submultiset. There are a couple algorithmic methods to compute the submultisets of size 3. The first would be to multiply out the polynomials $(1+x)(1+y+y^2)(1+z+z^2+z^3)$ and compute the degree 3 terms. The others are essentially equivalent, but drop the mention of polynomials, and I find the polynomials are a helpful computational aide.
Now because I only want the degree three terms, I'll only compute those.
We have $$[(1+x)(1+y+y^2)(1+z+z^2+z^3)]_3$$
$$=1[(1+y+y^2)(1+z+z^2+z^3)]_3 + x[(1+y+y^2)(1+z+z^2+z^3)]_2$$
$$=1(z^3+yz^2+y^2z) + x(z^2+yz+y^2)$$
$$=z^3+yz^2+y^2z+xz^2+xyz+xy^2,$$
corresponding to submultisets
$\newcommand{\multset}[1]{\{\{{#1}\}\}}\multset{3,3,3},\multset{2,3,3},\multset{2,2,3},\multset{1,3,3},\multset{1,2,3},$ and $\multset{1,2,2}$.
Now we can work out how many ways there are to order each multiset in the usual manner: the ways to order a multiset of the form $\multset{a,a,a}$ is $3!/3!=1$, $\multset{a,a,b}=3!/2!=3$, and $\multset{a,b,c}=3!=6$. Thus summing the number of ways to order each of our multisets, we get $1+3+3+3+6+3 = 19$ different ordered triples drawn from our multiset.
Best Answer
There are a total of 14! ways in which to arrange 14 marbles. However in this context, we would be overcounting. To see this we know there are 6! ways to arrange 6 red marbles. Similarly there are 4! ways to arrange the yellow marbles and 4! ways to arrange the green marbles. So we have over counted by a factor of 6!4!4!, as order of the positioning of marbles of the same color does not matter.