How can I find the number of partitions of $n$ into exactly $k$ distinct parts, where each part is at most $M$?
The number of partitions $p_k(\leq M,n)$ of $n$ into at most $k$ parts, each of size at most $M$, is given by the generating function:
$$
\binom{M+k}{k}_{x} = \prod_{j=1}^{k}\frac{1-x^{M+k-j+1}}{1-x^j}= \sum_{n=0}^{kM} p_{k}(\leq M,n) x^n
$$
For the number of the partitions $p_k(\mathcal{D},n)$ of $n$ into at most $k$ parts there is the recurrence relationship:
$$
p_{k}(\mathcal{D},n) = p_{k}(\mathcal{D},n-k) + p_{k-1}(\mathcal{D},n)
$$
But what, if I want to count only the partitions with distinct parts and restricted number of parts and restricted part size?
Update: Now I know the generating function for the number of distinct restricted partitions $p_k(\leq M, \mathcal{D},n)$ of $n$ into exactly $k$ distinct parts, all at most $M$ is
$$
\prod_{j=1}^{M} (1+xq^{j}) = \sum_{k,n=0}^{\infty}p_k(\leq M, \mathcal{D},n)x^{k}q^{n}
$$
and there is also a recurrence relation
$$
p_k(\leq M, \mathcal{D},n) =
p_{k-1}(\leq M-1, \mathcal{D},n-k) +
p_k(\leq M-1, \mathcal{D},n-k)
$$
How can I prove this? Could you recommend a book, where I could read about this?
Best Answer
In the "Update" (the day after the Question itself was posted), the OP mentions a recursion for counting the partitions of $n$ into $k$ distinct parts, each part at most $M$:
$$ p_k(\leq M, \mathcal{D},n) = p_{k-1}(\leq M-1, \mathcal{D},n-k) + p_k(\leq M-1, \mathcal{D},n-k) $$
and asks "How can I prove this?".
To see this, separate the required partitions on the basis of whether $1$ appears as a summand. If it does, then subtracting $1$ from each summand produces a partition of $n-k$ with exactly $k-1$ distinct parts (since the original summand $1$ disappears), each part at most $M-1$. These partitions are counted by the first term on the right-hand side of the recursion. Otherwise the summand $1$ does not appear, and subtracting $1$ from each part resulting in a partition of $n-k$ with exactly $k$ distinct parts, each part at most $M-1$. These cases are counted by the second term.
Note that a partition of $n$ with $k$ distinct parts exists if and only if $n \ge \binom{k+1}{2}$, because the ascending summands $m_1 + \ldots + m_k = n$ must satisfy $m_i \ge i$. If $n = \binom{k+1}{2}$, then there is just one such partition with $k$ distinct parts, the largest of which is $k$. Repeated application of the recursion will culminate with terms which we can evaluate "by inspection" as either zero or one.
By itself this recursion doesn't seem to give us an especially attractive way of evaluating $p_k(\leq M, \mathcal{D},n)$. Like the recursion for Fibonacci numbers, as a top-down method it suffers from recalculating terms multiple times (giving exponential complexity), so we would be better off working with it as a bottom-up method (giving polynomial complexity).
Better for large parameters is to close the circle with the ideas presented by @NikosM. by showing how the evaluation of $p_k(\leq M, \mathcal{D},n)$ can be reduced to counting restricted partitions without requiring distinct summands.
Prop. Suppose that $n \gt \binom{k+1}{2}$. Then the following are equal:
$(i)$ the number of partitions of $n$ into exactly $k$ distinct parts, each part at most $M$, i.e. $p_k(\leq M, \mathcal{D},n)$
$(ii)$ the number of partitions of $n - \binom{k}{2}$ into exactly $k$ parts, each part at most $M$
$(iii)$ the number of partitions of $n - \binom{k+1}{2}$ into at most $k$ parts, each part at most $M$
Sketch of proof: Once we know the ordered summands of partitions in $(i)$ satisfy $m_i \ge i$, it is easy to visualize their equivalents in $(ii)$ and $(iii)$ by Young tableaux, also called Ferrers diagrams. We remove a "base triangle" of dots corresponding to the first $i$ dots in the $i$th summand (since $m_i \ge i$) to get case $(iii)$, and remove one fewer dot in each summand to preserve exactly $k$ summands in case $(ii)$. These constructions are reversible, and the counts are equal.
Remark 1 If $n \le \binom{k+1}{2}$, $p_k(\leq M, \mathcal{D},n)=1$ if $n=\binom{k+1}{2}$ and $k \le M$ and otherwise $p_k(\leq M, \mathcal{D},n)=0$.
Remark 2 For fixed $k,n$, suppose $M_0 = n - \binom{k}{2} \gt 0$. Then for all $M \ge M_0$, $p_k(\leq M, \mathcal{D},n) = p_k(\leq M_0, \mathcal{D},n)$. That is, further increasing the upper bound $M$ on the size of parts will not yield additional partitions of $n$ with exactly $k$ distinct parts.
Recommended Reading
Andrews, George E. The Theory of Partitions (Cambridge University Press, 1998)
A modern classic for theory of integer partitions, reviewed by Richard Askey in BAMS.