Lets assume the non-trivial case $k < l \cdot m$. Each bin can receive a maximum of $l$ balls. I need distribute $k$ balls over $n$ bins so that some $m < n$ bins are occupied (not necessarily completely) and other $(n-m)$ remain empty. How many ways exist to do such distribution?
Combinatorics – Ways to Distribute Indistinguishable Balls in Bins
combinatorics
Related Solutions
Here's another way that seems a little slicker for the example, though I don't know which method scales up better.
Create a 4th bin for the balls you don't use. There are $4^5$ ways of putting the 5 balls into the 4 bins. Now use inclusion-exclusion; subtract the ones where one of the three bins is empty, add back in the ones where two bins are empty, subtract the ones where all three bins are empty. I get $$4^5-3\times 3^5+3\times2^5-1^5$$
EDIT: To relate this to well-studied concepts, the number of ways to put $n$ distinguishable balls into $m$ distinguishable bins, if you have to use all the balls and leave no bin empty, is $m!S(n,m)$, where $S(n,m)$ are the Stirling numbers of the second kind, q.v.
Now if you don't use all the balls, that's the same as putting the unused balls into an extra bin. So the answer to your problem is $$m!S(n,m)+(m+1)!S(n,m+1)$$
There is much information on Stirling numbers in books and online.
Applying $S_2$ here will be quite convoluted.
As explained in another question of yours,
The number of ways of getting exactly k different values when you throw n fair r-sided dice is
$\binom{r}{k}\cdot S_2(n,k)\cdot k! $
so to get exactly $6$ faces in $12$ throws of a normal die, $\binom66\cdot S_2(12,6)\cdot6!$ ways,
but this would include $11$ patterns, viz.
$\boxed 7\boxed 1\boxed 1\boxed 1\boxed 1\boxed 1\;\;\boxed 6\boxed 2\boxed 1\boxed 1\boxed 1\boxed 1\;\;\boxed 5\boxed 3\boxed 1\boxed 1\boxed 1\boxed 1\;\; \boxed 5\boxed 2\boxed 2\boxed 1\boxed 1\boxed 1\;\;\boxed 4\boxed 4\boxed 1\boxed 1\boxed 1\boxed 1$
$\boxed 4\boxed 3\boxed 2\boxed 1\boxed 1\boxed 1\;\;\boxed 4\boxed 2\boxed 2\boxed 2\boxed 1\boxed 1\;\;\boxed 3\boxed 3\boxed 3\boxed 1\boxed 1\boxed 1 \;\;\boxed 3\boxed 3\boxed 2\boxed 2\boxed 1\boxed 1\;\;\boxed 3\boxed 2\boxed 2\boxed 2\boxed 2\boxed 1$
$\boxed 2\boxed 2 \boxed 2 \boxed 2 \boxed 2 \boxed 2$
and we are interested only in the last pattern.
Now the number of ways for placing $12$ distinguishable balls in $6$ bins, the bins being indistinguishable except by occupancy, the multinomial coefficient has to be divided by ${p! q!..}$ where $p$ bins each have $p_n$ balls, $q$ bins each have $q_n$ balls, etc, e.g. for $\boxed 4\boxed 2\boxed 2\boxed 2\boxed 1\boxed 1$, there are $\binom{12}{4,2,2,2,1,1}/(3!2!)$ ways.
$S_2(12,6)= 1,323,652$ From this, if we subtract number of ways for $10$ unwanted patterns, we get
$1323652 - \left[\binom{12}{7,1,1,1,1,1}/ 5!+\binom{12}{6,2,1,1,1,1,}/4! ...+\binom{12}{5,2,2,1,1,1}/(2!3!) + ... \binom{12}{3,2,2,2,2,1}/4!\right] = 10,395$
$10,395$ is the number of ways to place$2$ each of $12$ distinguishable balls in $6$ indistinguishable bins,
So $10,395\times 6! = 7,484,400$, the desired answer.
But this looks like trying to catch your nose by bringing your hand from behind your neck !
Best Answer
First, you can select the non-empty bins in $\binom{m}{n}$ ways. So the remaining problem is to figure out how to distribute $k$ balls among $m$ bins win none getting more than $l$.
By stars and bars there are $\binom{k - 1}{m - 1}$ ways to distribute $k$ balls into $m$ bins with none empty. But this disregards the cases where some bins have more than $l$ balls. This in turn can be handled by the principle of inclusion and exclusion, considering the different sets as the solutions that violate 1, 2, ... of the constraints. Say the first bin has more than $l$ balls. Add $l$ balls to the first bin, you still have to distribute $k - l$ balls among all $m$ bins (overfilling bin number 1) so that no bin is empty, which can be done in $\binom{k - l - 1}{m - 1}$ ways. But the bin violating the constraint can be selected in $\binom{m}{1}$ ways. In all, adding in the factor given at the beginning: $$ \binom{n}{m} \sum_{j \ge 0} (-1)^j \binom{m}{j} \binom{k - j l - 1}{m - 1} $$