If the marbles are all distinct, then your analysis for the first problem is right. (If the marbles are identical, then the number of ways is $\binom{8}{4}$.)
Let's look at the second problem. We have $11$ identical apples. Line them up like this:
$$A\quad A \quad A\quad A \quad A \quad A \quad A \quad A \quad A\quad A \quad A,$$
with a gap between consecutive apples. There are $10$ interapple gaps. We will do our distribution as follows. We will choose $5$ of these gaps to put a separator (say a grape) into. Child $1$ will get all the apples from the leftmost one to the first grape. Child $2$ gets all the apples from the first grape to the second grape. Child $3$ will get all the apples from the second grape to the third, and so on. Finally, Child $6$ gets all the apples from the fifth grape to the right end.
There are exactly as many ways to distribute the apples between the kids as there are to place $5$ grapes as described above. For if we know how many apples each of the kids gets, we know where to place the grapes. Also, as we saw above, every grape placement determines what each child gets. It follows that the number of ways to distribute the apples is $\binom{10}{5}$.
Remark: For completeness, we deal with the marble problem on the assumption marbles are identical. The idea is similar to the apple one, but with a twist. We want to distribute $4$ marbles between $5$ pockets (children). What is different is that we are not asking that each child get a marble.
Imagine doing the distribution as follows. We will distribute $9$ marbles among $5$ children, so that each child gets at least one marble. Then we take away a marble from each child. We will end up with $4$ marbles distrbuted among $5$ children.
Use the separator idea. There are $\binom{8}{4}$ to distribute $9$ identical marbles between $5$ children, with each child getting at least one marble. So there are $\binom{8}{4}$ ways to distribute $4$ marbles among $5$ children.
Actually, with pockets and identical marbles, the numbers are small enough that we could count more crudely. We could put all the marbles in one pocket. There are $\binom{5}{1}$ ways to do this. We could put $4$ marbles in one pocket and $1$ in another. The lucky pocket can be chosen in $\binom{5}{1}$ ways, and for each of these, the pocket that will have one marble can be chosen in $\binom{4}{1}$ ways, for a total of $\binom{5}{1}\binom{4}{1}$. Continue.
$x^{20}$ should be $x^{17}$ in your question.
$$(x+x^2+\cdots+x^{17})^{4}$$
$$=(x+x^2+\cdots+x^{17})\times (x+x^2+\cdots+x^{17})\times (x+x^2+\cdots+x^{17})\times (x+x^2+\cdots+x^{17})$$
So, you choose $x^A, x^B, x^C, x^D$ from each $()$ from the left to the right.
Then, what you need is the number of a set $(A,B,C,D)$ such that
$$A+B+C+D=17\ \text{and}\ A,B,C,D\ge 1$$
Note that the latter is already satisfied.
Each represents the number of oranges which a child get. Then, imagine when you find the coefficient of $x^{17}$. You'll choose every pattern of $(A,B,C,D)$ such that $A+B+C+D=17$.
Hence, the coefficient of $x^{17}$ represents the ways you can distribute the oranges. I hope this is helpful.
Best Answer
For the first problem, the simplest answer is that both generating functions are correct for the desired coefficient. i.e.,
\begin{align*} &[x^{20}] (x^4+\dotsb+x^{20})(1+\dotsb+x^{20})(1+\dotsb+x^{20})\\ &= [x^{20}] (x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16}) \end{align*}
where $[x^n]$ denotes the coefficient extraction operator.
The reason for this is, for instance, the coefficient of $x^{17}$ is never used from the second product term due to the contribution of terms with power at least $4$ from the first. The difference between our generating functions is that certain cases, such as cases in which the first person selects $4$ marbles and the second selects $17$ are encoded in the first generating function but not in the second. Encoded or not, $20$ marbles cannot be reached from $21$ through selection of marbles in the third term, so $[x^{20}]$ does not count such cases.
In terms of the problem at hand, we note that we can encode more states that won't ultimately affect the extracted coefficient. For much the same reasoning as higher order coefficients like $[x^{17}]$ not being considered when extracting, we could equally well conclude
\begin{align*}[x^{20}] &(x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16})\\ &= [x^{20}] (x^4+\dotsb+x^{a})(1+\dotsb+x^{b})(1+\dotsb+x^{c}) \end{align*} for $a\geq 20$, $b\geq 16$, and $c\geq 16$.
In fact this holds formally if $a=b=c=\infty$:
\begin{align*} &[x^{20}] (x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16})\\ &= [x^{20}] (x^4+x^5+\dotsb)(1+x+\dotsb)(1+x+\dotsb) \end{align*}
This is useful for when we want to actually do the coefficient extraction,
\begin{align*} &(x^4+x^5+\dotsb)(1+x+\dotsb)(1+x+\dotsb)\\ &= x^4(1+x+\dotsb)^3 = \frac{x^4}{(1-x)^3} \end{align*}
is more manageable than \begin{align*}&(x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16})\\ &= x^4(1+\dotsb+x^{16})^3 = \frac{x^4(1-x^{17})^3}{(1-x)^3}. \end{align*}
On to the actual computation, given that $\frac{1}{(1-x)^n} = \sum_{k=0}^\infty \binom{n+k-1}{n-1} x^k$,
\begin{align*} [x^{20}] \frac{x^4}{(1-x)^3} &= [x^{20}] \sum_{k=0}^\infty \binom{k+2}{2} x^{k+4}\\ &= [x^{20}] \sum_{k=4}^\infty \binom{k-2}{2} x^{k} = \binom{18}{2}. \end{align*}
For the second problem, I believe they meant $(x^3+\dotsb+x^{20})^3$ as $(1+x+\dotsb+x^{20})^3$ encodes $1\cdot 1\cdot x^{20}$ which is not one of the valid cases.
At least $3$ marbles were taken by each, so you can just look at $(x^3+\dotsb+x^{14})^3$. Another way is to first reserve $3$ marbles per person, and then selecting up to $20-9=11$ marbles for each, which corresponds to the generating function $(x^3)^3(1+\dotsb+x^{11})^3$ (which you'll note is algebraically equivalent). All these generating functions have the same $x^{20}$ coefficient:
\begin{align*} &\;[x^{20}] (x^3+\dotsb+x^{20})^3\\ = &\;[x^{20}] (x^3+x^4+\dotsb+x^{14})^3\\ = &\;[x^{20}] (x^3+x^4+\dotsb)^3 = [x^{20}] \frac{x^9}{(1-x)^3}\\ = &\;[x^{20}] \sum_{k=0}^\infty \binom{k+2}{2} x^{k+9} = [x^{20}] \sum_{k=9}^\infty \binom{k-7}{2} x^{k}\\ = &\binom{13}{2}. \end{align*}