There are 101 terms in the first factor, of which 100 have a power of $x$ greater than 0. The second factor can be written:
$$
\sum_{k_1 + k_2 + k_3 = 101} 1^{k_1} \cdot x^{2 k_2} \cdot x^{k_3}
$$
(just imagine the product multiplied out) so you are asking how many sets of (k_1, k_2, k_3), all integers at least 0, are such that $k_1 + k_2 + k_3 = 101$ with $k_1 < 101$ (that one gives the term 1, there is just one of those).
Let's find out how many solutions $k_1 + k_2 + k_3 = 101$ has, this is like chopping a line of 101 $*$ into three pieces, say by separating with $|$ (this is called a stars and bars argument, for obvious reasons). But then you have a total of $101 + 2$ positions to be filled with 101 stars and 2 bars, that can be done in $\binom{101 + 2}{2} = 5253$ ways, of which you subtract 1 for $k_1 = 101$.
Combining your factors, you have 101 terms in the first factor, $5252$ with $x$ from the second, and the terms with $x$ in the product are the result of multiplying any term from the first factor with a term containing $x$ from the second, i.e., $101 \cdot 5252 = 53042$ (as long as no simpolifications happen).
A lower bound is that the result is a polynomial of degree $100 + 2 \cdot 101 = 302$, so there are at most $301$ terms with powers of $x$. But there are negative terms, so cancellation can/will happen, and you get less.
This is probably the wrong proof for you, but I will post it anyways. (requires calculus)
Note that $f(x)=(a+x)^n$ is an analytic function in $x$ for arbitrary $a,n$ since on its own, it is a power series with one term.
If it is an analytic function, then it should follow Taylor's theorem.
Now, if we take the expansion around $x=0$, we get
$$(a+x)^n=a^n+na^{n-1}x+\frac{n(n+1)}2a^{n-2}x^2+\dots$$
Since $f(0)=a^n$, $f'(0)=na^{n-1}$, $\dots f^{(k)}(0)=n(n+1)(n+2)\dots(n+k-1)a^{n-k}$
or
$$(a+x)^n=\sum_{k=0}^\infty\frac{n(n+1)(n+2)\dots(n+k-1)}{k!}a^{n-k}x^n$$
$$(a+x)^n=\sum_{k=0}^\infty\binom nka^{n-k}x^n$$
where $f'(x)$ is the first derivative of $f(x)$, $f''(x)$ the second derivative, etc. $f^{(k)}(x)$ is the $k$th derivative of $f(x)$.
Best Answer
To convert to a more familiar form, multiply and divide by $x^6$:
$$(x-1)^3 (1+2x^2)^6 x^{-6}$$
Now you're looking for the coefficient of $x^4$ in $(x-1)^3 (1+2x^2)^6$. You can get $x^4$ out of this product either by multiplying $x^2$ with $x^2$ or $1$ with $x^4$. Each of those coefficients should be easy to calculate, then you just add them.