In short, row reduced echelon form(RREF) of a matrix $A$ is such that
i) Every leading entry is 1
ii) Any nonzero rows are above zero rows
iii) any leading entry is strictly to the right of any leading entries above that row
iv) any other entry in a column containing a leading entry is 0 except for the leading entry.
So it does not have to be put in augmented matrix $[A|b]$ to get a RRE form. You are comparing RRE form of matrix $A$ and $[A|b]$.
To see why the statement is true, suppose that you put the matrix $[A|b]$ into RRE form, so you have a matrix E. If E contains a leading entry in its last column, in terms of system of equations, what does it say? And what is the condition for E to not have any leading entry in last column?
Note: If RRE form of $[A|b]$ does contain a leading entry, then it is different from that of $A$. Also, note that RRE form of $[A|b]$ is m by n+1 whereas that of $A$ is m by n.
Solve:
$x+y=1$
$x+y=2$
Then we have
$\
A =
\left( {\begin{array}{cc}
1 & 1 \\
1 & 1
\end{array} } \right)
$
$\
b =
\left( {\begin{array}{cc}
1 \\
2
\end{array} } \right)
$
and $Ax=b$
If we turn A into RREF, we get
$\
E =
\left( {\begin{array}{cc}
1 & 1 \\
0 & 0
\end{array} } \right)
$
So A has rank 1
and if we put $[A|b]$ into RRE form, we get
$\
E' =
\left( {\begin{array}{cc}
1 & 1 & 0 \\
0 & 0 & 1
\end{array} } \right)
$
So augmented matrix has rank 2. Observe what last row says in terms of equations.
The reduced row echelon form of a matrix is unique. The fact that the two given vectors form a basis of the null space means that the reduced form of the homogeneous linear system associated to the matrix is
$$
\begin{cases}
x_1=4x_2+x_4\\
x_3=2x_4
\end{cases}
$$
because, for $x_2=1$ and $x_4=0$ we get the first vector and with $x_2=0$ and $x_4=1$ we get the second vector. So the reduced system can be written
$$
\begin{cases}
x_1-4x_2-x_4=0\\
x_3-2x_4=0
\end{cases}
$$
which corresponds to the matrix
$$
\begin{bmatrix}
1 & -4 & 0 & -1 \\
0 & 0 & 1 & -2 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0
\end{bmatrix}
$$
Your error is considering the third and fourth variables as free, while they are the second and fourth ones.
Best Answer
The determinant is given by
$$\det A = \begin{vmatrix} 1 & 2 & -1 \\ \color{red}2 & \color{red}0 & \color{red}2\\ -1 & 2 & k \end {vmatrix} = -\color{red} 2\begin{vmatrix}2 & -1 \\ 2 & k \end {vmatrix} +\color{red} 0\begin{vmatrix}1 & -1 \\ -1 & k \end {vmatrix} -\color{red} 2\begin{vmatrix}1 & 2 \\ -1 & 2 \end {vmatrix} \\= -2(2k+2) - 2(2 + 2) = -4k - 12$$
Now $A$ is not invertible $\Leftrightarrow \det A = 0$. Then $k = -3$.