Let's define some $q$-series notation: $(a;q)_n = \prod_{i=0}^{n-1} (1 - aq^i)$ with shorthand notation $(q)_n = (q;q)_n$. E.g. the left-hand side of Euler's pentagonal theorem as stated in the question is $(x)_\infty$.
Now, $$P(n,k,m) = [x^n z^k] \prod_{i=1}^m \frac{1}{1 - x^i z} = [x^n z^k] \frac{\prod_{i=m}^\infty (1 - x^{i+1} z)}{\prod_{i=0}^\infty (1 - x^{i+1} z)} = [x^n z^k]
\frac{(x^{m+1}z;x)_\infty}{(xz;x)_\infty}$$
There are a couple of theorems due to Euler: $$(-x;q)_\infty = \sum_{k=0}^\infty q^{k(k-1)/2}(q)_k^{-1} x^k \\
(-x;q)_\infty^{-1} = \sum_{k=0}^\infty (-1)^k (q)_k^{-1} x^k$$
Then $$(x^{m+1}z;x)_\infty(xz;x)_\infty^{-1} = \left(\sum_{k=0}^\infty x^{k(k-1)/2}(x)_k^{-1} (-x^{m+1}z)^k\right) \left(\sum_{k=0}^\infty (-1)^k (x)_k^{-1} (-xz)^k\right) \\
= \left(\sum_{j=0}^\infty (-1)^j x^{j(j+2m+1)/2}(x)_j^{-1} z^j\right) \left(\sum_{k=0}^\infty (x)_k^{-1} x^k z^k\right) \\
= \sum_{j=0}^\infty \sum_{k=0}^\infty \frac{(-1)^j x^{j(j+2m+1)/2+k} z^{j+k}}{(x)_j (x)_k}$$
In particular, $$P(n,k,m) = [x^n] \sum_{j=0}^\infty \frac{(-1)^j x^{j(j+2m-1)/2+k}}{(x)_j (x)_{k-j}}$$
Note that if $P'(n,m)$ counts the partitions of $n$ into parts of at most $m$ then $$P'(n,m) = [x^n] \prod_{i=1}^m \frac{1}{1-x^i} = [x^n] (x)_m^{-1}$$ so $$P(n,k,m) = \sum_{a=0}^\infty \sum_{b=0}^\infty \sum_{j=0}^\infty (-1)^j P'(a,j) P'(b,k-j) [n-k-a-b=j(j+2m-1)/2]\\
= \sum_{a=0}^\infty \sum_{b=0}^\infty \sum_{j \in \frac{1-2m\pm\sqrt{4m^2-4m+1+8(n-k-a-b)}}{2}} (-1)^j P'(a,j) P'(b,k-j)$$
It's not quite as elegant as the pentagonal theorem...
Best Answer
Each of the $k$ parts must have a minimum of 1 item, leaving $n-k$ to distribute into the $k$ bins. When you choose how many to put in the first $k-1$ bins, the number going into the last bin is fixed. So you have $k-1$ choices, each of which is within the range $[0,n-k]$, so there are $n-k+1$ of them. This is a generous upper bound.