HINT...you have a choice of 4, 5 or 6 for the first place (from left to right).
You then have a choice of 8 digits for the second place (including 0 but excluding your first choice).
Then you have a choice of 7 digits for the third place.....
can you finish this?
We multiply the number of ways of arranging the rings by the number of ways of distributing them to the fingers of the woman's right hand.
There are $5!$ ways to arrange five distinct rings. We place them on the fingers of the right hand from the bottom ring on the thumb (or the first finger on which a ring appears) to the top ring on the pinky (or the last finger on which a ring appears).
Next, we count the number of admissible ways of distributing the five rings to the fingers of her right hand.
Let $x_1$ be the number of rings placed on the thumb, $x_2$ be the number of rings on the index finger, $x_3$ be the number of rings placed on the middle finger, $x_4$ be the number of rings placed on the ring finger, and $x_5$ be the number of rings placed on the pinky. Since there are a total of five rings
$$x_1 + x_2 + x_3 + x_4 + x_5 = 5 \tag{1}$$
Equation 1 is an equation in the nonnegative integers. A particular solution of equation 1 corresponds to the placement of $5 - 1 = 4$ addition signs in a row of five ones. For instance
$$+ 1 + 1 + 11 + 1$$
corresponds to the solution $x_1 = 0$, $x_2 = 1$, $x_3 = 1$, $x_4 = 2$, $x_5 = 1$. The number of solutions to equation 1 in the nonnegative integers is
$$\binom{5 + 5 - 1}{5 - 1} = \binom{9}{4}$$
since we must choose which four of the nine positions required for five ones and four addition signs will be filled with addition signs.
However, we have the restriction that at most three rings may be placed on any one finger. Notice that at most one finger could have more than three rings on it since $2 \cdot 4 = 8 > 5$. There are five ways to select the finger which has more than three rings on it. Suppose that finger is the pinky. Then $x_5' = x_5 - 4$ is a nonnegative integer. Substituting $x_5' + 4$ for $x_5$ in equation 1 yields
\begin{align*}
x_1 + x_2 + x_3 + x_4 + x_5' + 4 & = 5\\
x_1 + x_2 + x_3 + x_4 + x_5' & = 1 \tag{2}
\end{align*}
Equation 2 is an equation in the nonnegative integers with five solutions. Hence, there are
$$\binom{5}{1}\binom{1 + 5 - 1}{5 - 1} = \binom{5}{1}\binom{5}{4}$$
solutions of equation 1 that violate the restriction that at most three rings may be placed on one finger.
Therefore, there are
$$5!\left[\binom{9}{5} - \binom{5}{1}\binom{5}{4}\right]$$
ways to distribute five distinct rings to the five fingers of her right hand so that a maximum of three rings is placed on any finger.
Notice that it is only necessary to arrange the rings once. This is why your answer is too large.
Best Answer
In the first case, if you are assuming all five rings are worn, you're close; you just mixed up your $n=4$ and $k= 5$. For the first: we can look at this as the number of ways to sum 4 non-negative integers to equal $5$: $$\underbrace{f_1 + f_2 + f_3 + f_4}_{n = 4\;\text{fingers}} = \underbrace{5}_{5\; \text{rings}}$$ where the $f$'s stand for fingers (four of them), and the 5 is the number of rings. This gives us: $$\binom{4 + 5 -1}{5} = \binom{8}{5} = \dfrac{8!}{5!3!} = \dfrac{8\cdot 7\cdot 6}{3\cdot 2 \cdot 1} = 56\;\;\text{combinations}$$
In the second case, there is no assumptions being made by your method: counting every possible way of wearing no ring, one ring...5 rings, on any one or more fingers gives us $5^4$ ways of wearing distinct rings, or no ring at all.
If you want to require that all rings be worn, as in case one, then we multiply the result of $(1)$ by $k! = 5!$, which gives all permutations of the rings, as they are distinct, and can thus be rearranged: that gives us $$(5!)\binom{4 + 5 -1}{5} = (5!)\binom{8}{5} = (5!)\dfrac{8!}{5!3!} = (5!)\dfrac{8\cdot 7\cdot 6}{3\cdot 2 \cdot 1} = (5!)56\;\;$$