For starters the rank is $\geq 1$ as there always exist nonzero elements. The rank is also $\leq 2$, due to the shape of the matrix.
Suppose there were a point $(x_1,x_2)$ where the rank of the Jacobian is $1$. In that case the columns of the Jacobian would be linearly dependent. This will imply that the $2 \times 2$ minors $$\det \begin{pmatrix} x_2 & x_1 \\ 2x_1 & -1 \end{pmatrix},\;\det \begin{pmatrix} x_2 & x_1 \\ 1 & 3x_2^2 \end{pmatrix}$$ are both zero. Is it possible?
In short, row reduced echelon form(RREF) of a matrix $A$ is such that
i) Every leading entry is 1
ii) Any nonzero rows are above zero rows
iii) any leading entry is strictly to the right of any leading entries above that row
iv) any other entry in a column containing a leading entry is 0 except for the leading entry.
So it does not have to be put in augmented matrix $[A|b]$ to get a RRE form. You are comparing RRE form of matrix $A$ and $[A|b]$.
To see why the statement is true, suppose that you put the matrix $[A|b]$ into RRE form, so you have a matrix E. If E contains a leading entry in its last column, in terms of system of equations, what does it say? And what is the condition for E to not have any leading entry in last column?
Note: If RRE form of $[A|b]$ does contain a leading entry, then it is different from that of $A$. Also, note that RRE form of $[A|b]$ is m by n+1 whereas that of $A$ is m by n.
Solve:
$x+y=1$
$x+y=2$
Then we have
$\
A =
\left( {\begin{array}{cc}
1 & 1 \\
1 & 1
\end{array} } \right)
$
$\
b =
\left( {\begin{array}{cc}
1 \\
2
\end{array} } \right)
$
and $Ax=b$
If we turn A into RREF, we get
$\
E =
\left( {\begin{array}{cc}
1 & 1 \\
0 & 0
\end{array} } \right)
$
So A has rank 1
and if we put $[A|b]$ into RRE form, we get
$\
E' =
\left( {\begin{array}{cc}
1 & 1 & 0 \\
0 & 0 & 1
\end{array} } \right)
$
So augmented matrix has rank 2. Observe what last row says in terms of equations.
Best Answer
Be careful when row-reducing matrices that have variable entries. You need to examine cases in which you might have divided by zero separately. Be especially wary if you use automated solvers to do your row-reduction for you. Most of them don’t handle variable entries the way one might like and will happily spit out a full rank matrix even if some values of those variables actually produce a rank-deficient matrix. This is the case here: if you ask Mathematica to row-reduce your Jacobian, it will produce a full-rank matrix—in fact the one you have in your question—even though the Jacobian is obviously rank-one when $y=z=0$.
This problem can be solved without using row-reduction at all, which avoids all of these considerations (although row-reduction is pretty easy here). Your Jacobian is rank-deficient iff one row is a scalar multiple of the other, which can be expressed as $(2x,2y,2z)\times(2x-2,2y,0)=0$. This will give you three equations in $x$, $y$ and $z$, although one of them will be redundant. If you prefer not to take advantage of special features of $\mathbb R^3$, then you can instead examine the three $2\times2$ minors of the Jacobian: it is rank-deficient iff they all vanish. This condition in fact generates same three equations for your Jacobian as does the cross-product equation above.