We distribute the pens first.
There are two cases:
- The three pens are distributed to three people of the five people.
This can be done in $5 \cdot 4 \cdot 3$ ways since there are five ways of choosing the recipient of the first pen, four ways of choosing the recipient of the second pen, and three ways of choosing the recipient of the third pen.
That leaves two people who have not received a pen. Since each person receives at least one pen or pencil, we give each of those two people one of the $19$ identical pencils. There are now $17$ identical pencils to distribute to $5$ people. Let $x_k$ be the number of remaining pencils distributed to the $k$th person. Then
$$x_1 + x_2 + x_3 + x_4 + x_5 = 17 \tag{1}$$
This is an equation in the non-negative integers. The number of solutions of equation 1 is the number of ways four addition signs can be inserted into a row of $17$ ones, which is
$$\binom{17 + 4}{4} = \binom{21}{4}$$
since we must choose which four of the $21$ symbols (four addition signs and seventeen ones) will be addition signs. In this case, the number of ways of distributing three different pens and $19$ identical pencils to five people so that each person receives at least one pen or pencil is
$$5 \cdot 4 \cdot 3 \cdot \binom{21}{4}$$
- The three pens are distributed to two of the five people.
There are five ways of selecting one person to receive two of the three pens. There are $\binom{3}{2}$ ways of selecting which two of the three pens that person receives. There are four ways of selecting the person who receives the third pen. Hence, the number of ways of distributing the pens to two of the five people is
$$5 \cdot \binom{3}{2} \cdot 4$$
That leaves three people who have not received a pen. Since each person receives at least one pen or pencil, we give each of those three people one of the $19$ identical pencils. That leaves $16$ pencils to distribute to $5$ people. Let $x_k$ be the number of pencils distributed to the $k$th person. Then
$$x_1 + x_2 + x_3 + x_4 + x_5 = 16 \tag{2}$$
Equation 2 is an equation in the non-negative integers with
$$\binom{16 + 4}{4} = \binom{20}{4}$$
solutions. In this case, the number of ways of distributing the pens and pencils to the five people so that each person receives at least one pen or pencil is
$$5 \cdot \binom{3}{2} \cdot 4 \cdot \binom{20}{4}$$
In total, the number of ways of distributing three different pens and five identical pencils to five people so that no person receives more than two pens and so that each person receives at least one pen or pencil is
$$5 \cdot 4 \cdot 3 \cdot \binom{21}{4} + 5 \cdot \binom{3}{2} \cdot 4 \cdot \binom{20}{4} = 60\left[\binom{21}{4} + \binom{20}{4}\right]$$
Your calculation of the number of solutions that violate the condition that $x_1+x_2\le 20$ fails to take into account that if $x_1+x_2=m$, say, there are $m+1$ different possibilities for $x_1$ and $x_2$: $x_1$ can be anything from $0$ through $m$. For instance, when $m=21$, there are $22$ possibilities for $x_1$ and $x_2$, not just one.
Here’s a slightly expanded version of the book’s solution, which does take all of that into account.
We give Lucky $7$ balls to start with. Then we pick some $k\in\{0,1,\ldots,20\}$ and distribute $k$ balls between Alice and Eve; this can be done in $k+1$ ways. We’re left with $23-k$ balls to distribute between Lucky and the fourth person, and we can do this in $24-k$ ways. We can combine any of the distributions of $k$ balls between Alice and Eve with any of the $24-k$ distributions of the remaining balls between Lucky and the fourth person, so there must be altogether $(k+1)(24-k)$ ways to distribute the balls so that Lucky gets at least $7$ balls, and between them Alice and Eve get $k$ balls. Now we simply add up the numbers for the different possible values of $k$ to get $\sum_{k=0}^{20}(k+1)(24-k)$.
Best Answer
Lets say that this represents some distribution of balls to 4 people.
****|**|*****|****
I only have 15 balls in this picture, but you hopefully get the point
Now think of this not as balls and people but as stars and bars.
there are 18 objects in my picture (or 33 in your problem) 3 of which are bars.
The total number of ways to put $n$ objects in $m$ bins, then is ${n-1\choose m-1}$
Now we have some addional criteria. Alice and eve get no more than 20. And lucky gets at least 7.
Lets fist give 7 to Lucky. And then we can remove them from consideration... that leaves 23 balls to distribute.
${26\choose 3}=2600$
And then we need to remove the cases when where Alice and Eve get more than 20.
Suppose we give Alice and Eve 20. Lucky gets at least 7, that is 27 of 30 balls accounted for. There are cases. 21 balls to A+E, 22 balls to A+E, 23 balls to A+E.
${22\choose 1}\cdot{3\choose 1} + {23\choose 1}\cdot{2\choose 1} + {24\choose 1}\cdot{1\choose 1} = 136$
$2600-136=2464$