In short, row reduced echelon form(RREF) of a matrix $A$ is such that
i) Every leading entry is 1
ii) Any nonzero rows are above zero rows
iii) any leading entry is strictly to the right of any leading entries above that row
iv) any other entry in a column containing a leading entry is 0 except for the leading entry.
So it does not have to be put in augmented matrix $[A|b]$ to get a RRE form. You are comparing RRE form of matrix $A$ and $[A|b]$.
To see why the statement is true, suppose that you put the matrix $[A|b]$ into RRE form, so you have a matrix E. If E contains a leading entry in its last column, in terms of system of equations, what does it say? And what is the condition for E to not have any leading entry in last column?
Note: If RRE form of $[A|b]$ does contain a leading entry, then it is different from that of $A$. Also, note that RRE form of $[A|b]$ is m by n+1 whereas that of $A$ is m by n.
Solve:
$x+y=1$
$x+y=2$
Then we have
$\
A =
\left( {\begin{array}{cc}
1 & 1 \\
1 & 1
\end{array} } \right)
$
$\
b =
\left( {\begin{array}{cc}
1 \\
2
\end{array} } \right)
$
and $Ax=b$
If we turn A into RREF, we get
$\
E =
\left( {\begin{array}{cc}
1 & 1 \\
0 & 0
\end{array} } \right)
$
So A has rank 1
and if we put $[A|b]$ into RRE form, we get
$\
E' =
\left( {\begin{array}{cc}
1 & 1 & 0 \\
0 & 0 & 1
\end{array} } \right)
$
So augmented matrix has rank 2. Observe what last row says in terms of equations.
Here is why your argument is incomplete. One way for your reduced matrix to have rank 2 is to have row three all zeros, as you said. HOWEVER, there is another possibility: row 2 and 3 being multiple of each other. (With row 1 this is not a question b/c row 1 cannot be multiple of either row 2 or row 3 since we have 1 versus 0s on first column.
So let us look at this second possibility: we will have rank 2 if
$$
\frac{-7}{a-3} = \frac{5-b}{-1-b} \, .
$$
Which hopefully is the same equation as the one in the answer below reached by computing determinant.
Best Answer
There are many methods, one is to compute its characteristic polynomial.
$$\chi(t) = -t^3 + (x+2) t^2 + (x-1)^2 t - (x-1)^2 (2 + x)$$ Then we see we have rank $3$ when $x \neq 1, -2$, rank $2$ when $x = -2$ and rank $1$ when $x = 1$.