We should form two cases, one of 0 (because 0 can't be the first one) and one with all numbers 1-9, to find out how many numbers there are that use a digit 3 or more times. Then, we can subtract this from 90,000 (the number of five digit numbers). This will give us the desired number.
Case 1: 0 shows up more than 3 times.
When you think about it, the only way 0 shows up more than three times is if the number is 10,000; 20,000; 30,000; etc., up to 90,000. So we just have to add nine to the following case.
Case 2: Some number 1-9 shows up more than 3 times.
This is slightly more tricky. We start with 1, just to make things easy. But when you really start to think about it, 1's would have to fill 4/5 slots in the number for it to break the rules. Therefore, the numbers (if the 5th number is 2) would be as follows:
11112
11121
11211
12111
21111
and we can do the same thing for every other number, 0 and 2-9. However, 01111 does not count, so we have to make sure to get rid of that one as well. Therefore, there are 8 times 5, or 40 ways this can be done for the number 1 for 2-9, and 4 ways this can be done with 0. Overall, the number 1 can be made in 44 different ways. This is also true for 2, 3, 4, 5, 6, 7, 8, and 9 (try plugging them in above where 1 is now). This gives us 9*44 + 9 ways to break the rules, or 9*45 ways. This comes out to 405 ways.
Subtract this from 90,000 and you will see that there are 89,595 ways to accomplish your task.
The structure of the analysis is the same as yours. We count the bad strings, where some digit appears exactly twice.
We first count the strings where say the digit $0$ appears exactly twice. Where the $0$'s are can be chosen in $\binom{5}{2}$ ways. For each of these ways, the remaining $3$ places can be filled in $9^3$ ways.
So our first estimate of the number of bad strings is $(10)\binom{5}{2}(9^3)$.
But we have double-counted, for example, the strings where $0$ and $1$ appear exactly twice. There are $\binom{5}{2}$ ways to choose where the $0$'s go, and for each there are $\binom{3}{2}$ ways to choose where the $1$'s go. The remaining spot can be filled in $8$ ways. Multiply by $\binom{10}{2}$ for the number of ways to choose the two digits that appear twice.
Putting things together, we find that the number of bad strings is
$$(10)\binom{5}{2}(9^3)-\binom{10}{2}\binom{5}{2}\binom{3}{2}(8).\tag{1}$$
Finally, subtract (1) from $10^5$.
Best Answer
Instead of dividing into cases, we use the Principle of Inclusion/Exclusion.
There are $4^6$ strings of length $6$ over our $4$-letter alphabet. Now we count the bad strings, in which one or more digits are missing.
There are $3^6$ strings with the digit $1$ mising, and also $3^6$ with $2$ missing, with $3$ missing, with $4$ missing.
So our first estimate for the number of bad strings is $4\cdot 3^6$.
However, when we added, we multiply counted the bad strings that have more than one missing digit. For example, there are $2^6$ strings with $1$ and $2$ missing, and the same for all $6$ ways to choose $2$ digits from $4$.
So our next estimate for the number of bad strings is $4\cdot 3^6-6\cdot 2^6$.
However, we have subtracted too much, for we have subtracted one too many times the $4$ strings with $3$ digits missing. So the number of bads is $4\cdot 3^6-6\cdot 2^6+4$.
More neatly, we can write the number of good strings as $$\binom{4}{0}4^6-\binom{4}{1}3^6+\binom{4}{2}2^6-\binom{4}{3}1^6.$$
The method readily generalizes.