You're given the system of linear equations described by
$$\mathbf{A}\mathbf{x}=\mathbf{b}\iff\begin{bmatrix}1&2&-3\\3&-1&5\\4&1&a^2-14\end{bmatrix}\begin{bmatrix}x\\y\\z\end{bmatrix}=\begin{bmatrix}4\\2\\a+2\end{bmatrix}$$
Your elimination procedure for writing $$\left[\begin{array}{ccc|c}1&2&-3&4\\3&-1&5&2\\4&1&a^2-14&a+2\end{array}\right]\stackrel{\text{elim}}{\implies}\left[\begin{array}{ccc|c}1&2&-3&4\\0&1&-2&\frac{10}{7}\\0&0&a^2-16&a-4\end{array}\right]$$
is correct.
Let's first assume $a=4$. Under this assumption, the last row is composed entirely of zeros and the matrix $\mathbf{A}$ is of rank $2$ (only two pivots in the $(1,1)$ and $(2,2)$ positions). This means the dimension of the nullspace (the set of vectors $\mathbf{x}$ such that $\mathbf{A}\mathbf{x}=\mathbf{0}$) of $\mathbf{A}$ is $1$, which means that, whether or not $\mathbf{A}\mathbf{x}=\mathbf{b}$ has a "special" solution that is a linear combination of the columns of $\mathbf{A}$, there will be a host (read: infinite number) of solutions determined by the vector in the basis of the nullspace.
With $a=4$, write the augmented matrix in RREF to obtain
$$\left[\begin{array}{ccc|c}1&0&1&0\\0&1&-2&0\\0&0&0&0\end{array}\right]$$
Because you know the basis of the nullspace contains one vector, you know that you have one free variable; that is, you can fix any of $x$, $y$, or $z$ and uniquely determine the value of the other two. Suppose $z=1$; then $x=-1$ and $y=2$. Therefore $\mathbf{x}=\begin{bmatrix}-1\\2\\1\end{bmatrix}$ is in the nullspace of $\mathbf{A}$, and furthermore any scalar multiple of this vector is also in the nullspace. This alone is enough to conclude that there is an infinite number of solutions to $\mathbf{A}\mathbf{x}=\mathbf{b}$ precisely when $a=4$.
Now suppose $a=-4$. The augmented matrix becomes
$$\left[\begin{array}{ccc|c}1&2&-3&4\\0&1&-2&\frac{10}{7}\\0&0&0&-8\end{array}\right]$$
The last row is problematic. There is no $\mathbf{x}$ that satisfies this equation.
Now assume $a\neq\pm4$. You correctly note that for this case, you can multiply the last row by $\frac{1}{a-4}$ to yield
$$\left[\begin{array}{ccc|c}1&2&-3&4\\0&1&-2&\frac{10}{7}\\0&0&a+4&1\\\end{array}\right]$$
Because there are three pivots (all sitting along the diagonal), the nullspace has dimension $0$ (so it only contains the zero vector) which means that if there is a solution to $\mathbf{A}\mathbf{x}=\mathbf{b}$, then $\mathbf{b}$ is exactly a linear combination of the columns of $\mathbf{A}$. The last row tells you when this happens. Solving, you get
$$\mathbf{x}=\begin{bmatrix}4-2\left(\frac{10}{7}+\frac{2}{a+4}\right)+\frac{3}{a+4}\\\frac{10}{7}+\frac{2}{a+4}\\\frac{1}{a+4}\end{bmatrix}=\begin{bmatrix}\frac{8}{7}-\frac{1}{a+4}\\\frac{10}{7}+\frac{2}{a+4}\\\frac{1}{a+4}\end{bmatrix}$$
To summarize:
There are infinitely many solutions when $a=4$.
There are no solutions when $a=-4$.
There is exactly one solution otherwise.
To answer your question about the "legal" move - that's absolutely fine so long as $a\neq4$. The problem with your work crops up just below that part: in the case of $a\neq\pm4$, when considering the last row you do not have the equation $a+4=1$, but rather $(a+4)z=1$.
To answer the more general question of when a matrix has no solution:
A system of equations can have one of three things: a unique solution, infinitely many solutions, and no solution.
Case One: unique solution
An augmented matrix has an unique solution when the equations are all consistent and the number of variables is equal to the number of rows. Simply put if the non-augmented matrix has a nonzero determinant, then it has a solution given by $\vec x = A^{-1}\vec b$.
Case Two: Infinitely many solutions
The number of rows is less than the number of variables. Think of it this way, we need one equation to solve for one unknown variable, two equations to solve for two variables, three equations to solve for three variables, and so on...
The number of rows represents(at most) the number of independent equations we have so if it's less than the number of columns which represents the number of variables, we most likely have the case of infinite solutions. Why infinite solutions? We cannot nail down at least of the variables to a value so we let it take on any real value.
Note: The non-augmented matrix must have determinant zero for this to be the case
Case Three: No Solutions
We have no solutions when the equations(represented by the rows) make no sense aka they are inconsistent. For example, when we put our augmented matrix in row reduced echelon form we have a non-zero value in the augmented column and zeroes every else. How can this be? It can't. A sum of all zeros is always zero.
Best Answer
It looks like you made a couple of mistakes in your working, and the reduced row echelon form of the left-hand matrix is actually the $3 \times 3$ identity matrix.
With that said, if you have an $n \times n$ reduced row echelon form with a bottom row that is all zeros to the left of the bar then you either have infinitely many solutions or none. Having exactly one solution is only possible if your RREF is the identity matrix.