Suppose that you use $p$ pennies, $n$ nickels, $d$ dimes, and $q$ quarters; then $p,n,d$, and $q$ must satisfy the system
$$\left\{\begin{align*}
&p+n+d+q=50\\
&p+5n+10d+25q=150\;.
\end{align*}\right.\tag{1}$$
Thus, you want to count the solutions to $(1)$ in non-negative integers.
The problem is small enough that you can easily solve by hand mostly using brute force. First, subtracting the first equation in $(1)$ from the second gives us the condition $$4n+9d+24q=100\;,\tag{2}$$
from which it’s clear that $q\le 4$.
Suppose that $q=4$; then $4n+9d=100-24\cdot4=4$, which clearly has only the solution $n=1,d=0$. At this point we’ve used four quarters, no dimes, and one nickel, for a total of $105$ cents from $5$ coins, and setting $p=45$ clearly gives the unique solution with $q=4$.
Now suppose that $q=3$, so that $(2)$ reduces to $4n+9d=28$. Clearly $d\le 3$. But both $4n$ and $28$ are divisible by $4$, while $9d$ is not for $0<d\le 3$, so we must have $d=0$ and $n=7$. We’ve now used $10$ coins that total $110$ cents, and it’s clear that setting $p=40$ gives us the unique solution with $q=3$.
Now suppose that $q=2$, so that $(2)$ reduces to $4n+9d=52$. Then $d\le 5$, and we can argue as in the previous case that $9d$ must be a multiple of $4$, so either $d=0$ and $n=13$, or $d=4$ and $n=4$. In the first case we have $15$ coins that total $115$ cents, and we must set $p=35$; in the second we have $10$ coins that total $110$ cents, and we must set $p=40$. These are clearly the only solutions when $q=2$.
If $q=1$, $(2)$ reduces to $4n+9d=76$, so that $d\le 8$. As in the previous two cases we must have $4\mid d$, so $d$ must be $0,4$, or $8$. These lead to the following solutions: $q=1,d=0,n=19,p=30$; $q=1,d=4,n=10,p=35$; and $q=1,d=8,n=1,p=40$.
Finally, if $q=0$, $(2)$ becomes $4n+9d=100$, the feasible values of $d$ are again $0,4$, and $8$, and the resulting solutions are: $q=0,d=0,n=25,p=25$; $q=0,d=4,n=16,p=30$; and $q=0,d=8,n=7,p=35$.
Thus, there are precisely $10$ solutions, and we’ve not just counted them, but also found them.
Best Answer
Your solution for $(1)$ is correct, and you can apply the same ideas to $(2)$.
First choose $3$ nickels, then choose the remaining $7$ coins, in $\binom{4+7-1}7=\binom{10}7=120$ ways. But there should be no more than $2$ quarters, so we need to subtract the choices with more than $2$ quarters. First choose $3$ nickels and $3$ quarters, then choose the remaining $4$ coins, in $\binom{4+4-1}4=\binom74=35$ ways. So the desired number is $120-35=85$.