Suppose that you use $p$ pennies, $n$ nickels, $d$ dimes, and $q$ quarters; then $p,n,d$, and $q$ must satisfy the system
$$\left\{\begin{align*}
&p+n+d+q=50\\
&p+5n+10d+25q=150\;.
\end{align*}\right.\tag{1}$$
Thus, you want to count the solutions to $(1)$ in non-negative integers.
The problem is small enough that you can easily solve by hand mostly using brute force. First, subtracting the first equation in $(1)$ from the second gives us the condition $$4n+9d+24q=100\;,\tag{2}$$
from which it’s clear that $q\le 4$.
Suppose that $q=4$; then $4n+9d=100-24\cdot4=4$, which clearly has only the solution $n=1,d=0$. At this point we’ve used four quarters, no dimes, and one nickel, for a total of $105$ cents from $5$ coins, and setting $p=45$ clearly gives the unique solution with $q=4$.
Now suppose that $q=3$, so that $(2)$ reduces to $4n+9d=28$. Clearly $d\le 3$. But both $4n$ and $28$ are divisible by $4$, while $9d$ is not for $0<d\le 3$, so we must have $d=0$ and $n=7$. We’ve now used $10$ coins that total $110$ cents, and it’s clear that setting $p=40$ gives us the unique solution with $q=3$.
Now suppose that $q=2$, so that $(2)$ reduces to $4n+9d=52$. Then $d\le 5$, and we can argue as in the previous case that $9d$ must be a multiple of $4$, so either $d=0$ and $n=13$, or $d=4$ and $n=4$. In the first case we have $15$ coins that total $115$ cents, and we must set $p=35$; in the second we have $10$ coins that total $110$ cents, and we must set $p=40$. These are clearly the only solutions when $q=2$.
If $q=1$, $(2)$ reduces to $4n+9d=76$, so that $d\le 8$. As in the previous two cases we must have $4\mid d$, so $d$ must be $0,4$, or $8$. These lead to the following solutions: $q=1,d=0,n=19,p=30$; $q=1,d=4,n=10,p=35$; and $q=1,d=8,n=1,p=40$.
Finally, if $q=0$, $(2)$ becomes $4n+9d=100$, the feasible values of $d$ are again $0,4$, and $8$, and the resulting solutions are: $q=0,d=0,n=25,p=25$; $q=0,d=4,n=16,p=30$; and $q=0,d=8,n=7,p=35$.
Thus, there are precisely $10$ solutions, and we’ve not just counted them, but also found them.
Here's some intuition:
Suppose you have 15 coins as follows:
O O O O O O O O O O O O O O O O
How many ways can you divide up these 15 coins into 4 different categories of coins? For instance:
O | O O | O O O O O O O O O | O O O
(Pennies | Nickels | Dimes | Quarters)
Hope this is helpful, let me know if you need another hint :)
Best Answer
Here is one approach. Let $d$ represent the number of dimes you choose. This number is distinguished from the other two, because it can't go past $10$, but the other two can go up to $16$. Let $n$ represent the number of nickels. With $d$ dimes chosen, you can have $0$ through $16-d$ nickels. Then the number of pennies is determined as $16-d-n$.
$$\begin{align} \sum_{d=0}^{10}\sum_{n=0}^{16-d}1&=\sum_{d=0}^{10}(17-d)\\ &=17\cdot11-\binom{11}{2}\\ &=132 \end{align}$$
Alternatively,
$$\begin{align} \sum_{d=0}^{10}\sum_{n=0}^{16-d}1&=\sum_{d=0}^{10}(17-d)=17+16+\cdots+7\\ &=\sum_{d=1}^{17}d-\sum_{d=1}^6d\\ &=\binom{18}{2}-\binom{7}{2}\\ &=132 \end{align}$$