Perhaps we should begin by discussing the proof of the proposition in detail. If we emphasize a particularly important step, we will then see why when we consider $\mathbb{Z}[x,y]$, we are lead to the remark that you're asking about.
The proof of the proposition (paraphrasing):
Assume $p(x)$ is reducible in $R[x]$ but irreducible in $(R/I)[x]$.
Then $p(x)=f(x)g(x)$ with $f(x), g(x)$ monic, non-constant polynomials. Then
reducing $f(x)$ and $g(x)$ mod $I$ gives a factorization in
$(R/I)[x]$. Contradiction.
Notice that it was critical that both $f(x)$ and $g(x)$ were monic (or at least that their leading coefficients are units, which can be shown to be equivalent to choosing them to be monic). Why? Because otherwise $f(x)$ could reduce to a unit in $(R/I)[x]$ (this could happen easily, think of $3x+1 \in \mathbb{Z}[x]$ and then consider $\mathbb{Z}_3[x]$). Then we wouldn't have found found that $p(x)$ was reducible in $(R/I)[x]$ and there would be no contradiction!
For completeness: the key point in this proposition is that we are assuming that $p(x)$ is monic and $I$ is a proper ideal. Then the leading coefficients of $f(x)$ and $g(x)$ are units (we argue that we can always multiply by a constant to arrange the leading coefficients to be $1$) and since $I$ is a proper ideal, these coefficients are not in $I$. Then the leading coefficients of $f(x)$ mod $I$ and $g(x)$ mod $I$ are not zero and so we have a nontrivial factorization in $(R/I)[x]$.
So really we need to emphasize why $f(x)$ and $g(x)$ couldn't reduce to a unit. Then it becomes more clear why we need to also worry about factors reducing to units when we consider polynomial rings of several variables.
Some further remarks:
Let's say we try to extend this proposition to $R[x,y]$. But we immediately have a problem: we know we need some kind of "leading coefficient" condition if we hope to repeat the proof as before (or else we need some other condition and strategy to show that reducible $q(x,y)$ cannot become irreducible in a quotient by a proper ideal!). But what we mean by "leading coefficient" is unclear for something like
\begin{align}
q(x,y) = x^3y-6xy^3+x^2y^2+x^2y+3xy^2+x-2y+1
\end{align}
Maybe we mean coefficients of terms of highest degree ($x^3y-6xy^3+x^2y^2$ in this case). Then is it enough to ask that at least one of them have coefficient $1$? All of them? But then notice that
\begin{align}
q(x,y) = (x^2y+3xy^2+1)(x-2y+1)
\end{align}
and so one of the maximum degree terms $x^2y^2$ doesn't arise as simply the product of two lower degree terms - it is the sum of two products. So $x^2y^2$ having coefficient $1$ didn't mean that it couldn't come from terms that are $0$ mod some ideal. The situation can get worse for higher degrees and higher numbers of variables: terms with coefficient 1 could come from a large number of sums of terms. This means we have to think carefully before trying to apply our previous proof.
One such strategy is to recognize that $R[x,y] \cong (R[x])[y]$. In other words the multivariable case can be thought of as the single variable case with coefficients coming from the ring $R[x]$ and so then the condition is simply that the term with the highest power of $x$ (or $y$) should be monic (more generally: check that it's enough to require that the coefficient of the highest power of say $y$ is not an element of the ideal $I[x] \subset R[x]$ that you want to reduce by). If this is the case, then we can simply apply the original proposition of the single variable case. For example
\begin{align}
p(x,y) = y^2 + (7x^2+14x)y+17
\end{align}
has this property and so we have $p(x,y) = y^2 + 3$ mod $7$, which is irreducible and so $p(x,y)$ was irreducible.
Notice that Dummit and Foote's example of $xy+x+y+1$ does not meet this criteria (neither the coefficient of the highest power of $x$ nor $y$ is $1$), so while this wouldn't have proved it was reducible, it would have prevented us from making the mistake they warn against. This criteria also would have been applicable to their other example $x^2 + xy+1$ (and we would have found that it is irreducible).
Since they have not yet introduced the idea of reducing mod some $I[x,y]$, hopefully combining the two parts of my answer fully resolves what we're trying to do, why we can do it, and where we can't, at least with respect to the context of what they're trying to point out at this point in the book.
That common definition of an (ir)reducible polynomial works over a field but needs to be modified when you pass to more general coefficient rings, because they may contain constants that are nonzero nonunits. In your example, $\,f = 2x^2+4\,$ is irreducible over $\,\Bbb Q,\,$ but is reducible over $\,\Bbb Z\,$ where $\,f = 2(x^2+2)\,$ and both factors are nonzero nonunits, but both factors are not of lower degree.
Over non-fields, the constants can convey very useful divisibility information so we don't want to ignore them when studying polynomial divisibility. For example, one way to view Gauss's Lemma is that the prime $\,p\in\Bbb Z\,$ extends to a prime in $\,\Bbb Z[x],\,$ i.e. $\,p\mid fg\,\Rightarrow\,p\mid f\,$ or $\,p\mid g.\,$
Best Answer
It is not true that if $p$ is reducible then it is a unit. Recall that an element $p\in F[X]$ is a unit if there exists $q\in F[X]$ such that $pq=1$. If $F$ is a field, then comparing degrees shows that all units are nonzero constants. Conversely all nonzero constants are clearly units.
Now consider for example the polynomial $p=X^2\in F[X]$, which is not a unit. Then for the polynomials $f=g=X\in F[X]$ we have $p=fg$, and neither $f$ nor $g$ is a unit. This means $p$ is reducible, but not a unit.
The same reasoning shows that more generally, a product of two nonconstant polynomials in $F[X]$ is reducible but not a unit.