Suppose that you use $p$ pennies, $n$ nickels, $d$ dimes, and $q$ quarters; then $p,n,d$, and $q$ must satisfy the system
$$\left\{\begin{align*}
&p+n+d+q=50\\
&p+5n+10d+25q=150\;.
\end{align*}\right.\tag{1}$$
Thus, you want to count the solutions to $(1)$ in non-negative integers.
The problem is small enough that you can easily solve by hand mostly using brute force. First, subtracting the first equation in $(1)$ from the second gives us the condition $$4n+9d+24q=100\;,\tag{2}$$
from which it’s clear that $q\le 4$.
Suppose that $q=4$; then $4n+9d=100-24\cdot4=4$, which clearly has only the solution $n=1,d=0$. At this point we’ve used four quarters, no dimes, and one nickel, for a total of $105$ cents from $5$ coins, and setting $p=45$ clearly gives the unique solution with $q=4$.
Now suppose that $q=3$, so that $(2)$ reduces to $4n+9d=28$. Clearly $d\le 3$. But both $4n$ and $28$ are divisible by $4$, while $9d$ is not for $0<d\le 3$, so we must have $d=0$ and $n=7$. We’ve now used $10$ coins that total $110$ cents, and it’s clear that setting $p=40$ gives us the unique solution with $q=3$.
Now suppose that $q=2$, so that $(2)$ reduces to $4n+9d=52$. Then $d\le 5$, and we can argue as in the previous case that $9d$ must be a multiple of $4$, so either $d=0$ and $n=13$, or $d=4$ and $n=4$. In the first case we have $15$ coins that total $115$ cents, and we must set $p=35$; in the second we have $10$ coins that total $110$ cents, and we must set $p=40$. These are clearly the only solutions when $q=2$.
If $q=1$, $(2)$ reduces to $4n+9d=76$, so that $d\le 8$. As in the previous two cases we must have $4\mid d$, so $d$ must be $0,4$, or $8$. These lead to the following solutions: $q=1,d=0,n=19,p=30$; $q=1,d=4,n=10,p=35$; and $q=1,d=8,n=1,p=40$.
Finally, if $q=0$, $(2)$ becomes $4n+9d=100$, the feasible values of $d$ are again $0,4$, and $8$, and the resulting solutions are: $q=0,d=0,n=25,p=25$; $q=0,d=4,n=16,p=30$; and $q=0,d=8,n=7,p=35$.
Thus, there are precisely $10$ solutions, and we’ve not just counted them, but also found them.
Best Answer
If we let $p$, $n$, $d$, and $q$ denote, respectively, the number of pennies, nickels, dimes, and quarters contained in the collection, then $$p + n + d + q = 30 \tag{1}$$ A particular solution equation 1 corresponds to the placement of three addition signs in a row of $30$ ones. For instance, $$+ 1 1 1 1 1 + 1 1 1 1 1 1 1 1 1 1 + 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1$$ corresponds to the solution $p = 0$, $n = 5$, $d = 10$, and $q = 15$. The number of such solutions is the number of ways we can place three addition signs in a row of thirty ones, which is $$\binom{30 + 4 - 1}{4 - 1} = \binom{33}{3} = \binom{33}{30} = \frac{33!}{30!3!}$$ since we must choose which three of the thirty-three positions required for thirty ones and three addition signs will be filled with addition signs or, equivalently, which thirty of the thirty-three positions positions required for thirty ones and three addition signs will be filled with ones.
This appears to be what you had in mind. However, you did not use parentheses correctly in your answer.
$$\binom{30 + 4 - 1}{30} = \frac{(30 + 4 - 1)!}{30!(4 - 1)!} = \frac{33!}{30!3!}$$
We must subtract those collections which include at least $16$ quarters from the total. Suppose $q \geq 16$. Then $q' = q - 16$ is a nonnegative integer. Substituting $q' + 16$ for $q$ in equation 1 yields \begin{align*} p + n + d + q' + 16 & = 30\\ p + n + d + q' & = 14 \tag{2} \end{align*} Equation 2 is an equation in the nonnegative integers with $$\binom{14 + 4 - 1}{4 - 1} = \binom{17}{3}$$ solutions.
Hence, the number of collections of $30$ coins that can be formed with at most $15$ quarters is $$\binom{33}{3} - \binom{17}{3}$$