For the first problem, the simplest answer is that both generating functions are correct for the desired coefficient. i.e.,
\begin{align*}
&[x^{20}] (x^4+\dotsb+x^{20})(1+\dotsb+x^{20})(1+\dotsb+x^{20})\\
&= [x^{20}] (x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16})
\end{align*}
where $[x^n]$ denotes the coefficient extraction operator.
The reason for this is, for instance, the coefficient of $x^{17}$ is never used from the second product term due to the contribution of terms with power at least $4$ from the first. The difference between our generating functions is that certain cases, such as cases in which the first person selects $4$ marbles and the second selects $17$ are encoded in the first generating function but not in the second. Encoded or not, $20$ marbles cannot be reached from $21$ through selection of marbles in the third term, so $[x^{20}]$ does not count such cases.
In terms of the problem at hand, we note that we can encode more states that won't ultimately affect the extracted coefficient. For much the same reasoning as higher order coefficients like $[x^{17}]$ not being considered when extracting, we could equally well conclude
\begin{align*}[x^{20}] &(x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16})\\
&= [x^{20}] (x^4+\dotsb+x^{a})(1+\dotsb+x^{b})(1+\dotsb+x^{c})
\end{align*}
for $a\geq 20$, $b\geq 16$, and $c\geq 16$.
In fact this holds formally if $a=b=c=\infty$:
\begin{align*}
&[x^{20}] (x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16})\\
&= [x^{20}] (x^4+x^5+\dotsb)(1+x+\dotsb)(1+x+\dotsb)
\end{align*}
This is useful for when we want to actually do the coefficient extraction,
\begin{align*}
&(x^4+x^5+\dotsb)(1+x+\dotsb)(1+x+\dotsb)\\
&= x^4(1+x+\dotsb)^3
= \frac{x^4}{(1-x)^3}
\end{align*}
is more manageable than
\begin{align*}&(x^4+\dotsb+x^{20})(1+\dotsb+x^{16})(1+\dotsb+x^{16})\\
&= x^4(1+\dotsb+x^{16})^3
= \frac{x^4(1-x^{17})^3}{(1-x)^3}.
\end{align*}
On to the actual computation, given that $\frac{1}{(1-x)^n} = \sum_{k=0}^\infty \binom{n+k-1}{n-1} x^k$,
\begin{align*}
[x^{20}] \frac{x^4}{(1-x)^3}
&= [x^{20}] \sum_{k=0}^\infty \binom{k+2}{2} x^{k+4}\\
&= [x^{20}] \sum_{k=4}^\infty \binom{k-2}{2} x^{k}
= \binom{18}{2}.
\end{align*}
For the second problem, I believe they meant $(x^3+\dotsb+x^{20})^3$ as $(1+x+\dotsb+x^{20})^3$ encodes $1\cdot 1\cdot x^{20}$ which is not one of the valid cases.
At least $3$ marbles were taken by each, so you can just look at $(x^3+\dotsb+x^{14})^3$. Another way is to first reserve $3$ marbles per person, and then selecting up to $20-9=11$ marbles for each, which corresponds to the generating function $(x^3)^3(1+\dotsb+x^{11})^3$ (which you'll note is algebraically equivalent). All these generating functions have the same $x^{20}$ coefficient:
\begin{align*}
&\;[x^{20}] (x^3+\dotsb+x^{20})^3\\
= &\;[x^{20}] (x^3+x^4+\dotsb+x^{14})^3\\
= &\;[x^{20}] (x^3+x^4+\dotsb)^3
= [x^{20}] \frac{x^9}{(1-x)^3}\\
= &\;[x^{20}] \sum_{k=0}^\infty \binom{k+2}{2} x^{k+9}
= [x^{20}] \sum_{k=9}^\infty \binom{k-7}{2} x^{k}\\
= &\binom{13}{2}.
\end{align*}
Best Answer
For $\bf{I_1}$ and $\bf{I_2}$, note that $$ \frac{{1 + w}}{{1 - z(1 + w)}} = \sum\limits_{m = 0}^\infty {z^m (1 + w)^{m + 1} } = \sum\limits_{m,n = 0}^\infty \binom{m + 1}{n}z^m w^n $$ provided $|z(1+w)|<1$. We can take, for instance, $R=\frac{1}{2}$ and $S=1$. From this you can see that $g_{0,0} = 0$, $g_{0,1} = 0 \ne 1$ and $g_{m,n} > 0$ if $m + 1 \ge n \ge 2$.
For $\bf{I_3}$, you can check that $r = 3 - 2\sqrt 2$ and $s = \frac{{\sqrt 2 }}{2}$ satisfy the requirements. (In fact, they are the unique positive solutions of the characteristic system.)
I leave you as an exercise to show, via Theorem VII.$\bf 3$, that $$ D_n=[z^n ]D(z) = \frac{{\sqrt 2 + 1}}{{2^{7/4} \sqrt \pi \, n^{3/2} }}(3 + 2\sqrt 2 )^n \!\left( {1 + \mathcal{O}\!\left( {\frac{1}{n}} \right)} \right), $$ as $n\to +\infty$.
Remark. The generating function has an explicit expression: $$ D(z) = \frac{{1 - 3z - \sqrt {z^2 - 6z + 1} }}{{4z}}. $$ You can do direct singularity analysis to $D(z)$ to obtain the asymptotics for $D_n$.