I do not know of any name for the operation so I will try and make an argument for why it could be called "reflected transpose".
Consider the $3\times 3$ matrix :
$${\bf A} = \left[\begin{array}{ccc}1&2&3\\4&5&6\\7&8&9\end{array}\right]$$
using lexical vectorization, we get:
$$\mathrm{vec}({\bf A}) = \left[\begin{array}{ccccccccc}
1&4&7&2&5&8&3&6&9
\end{array}\right]^T$$
We can now define this "flipped identity" reflection in the vectorization:
$${\bf R} = \left[\begin{array}{ccccccccc}0&0&0&0&0&0&0&0&1\\0&0&0&0&0&0&0&1&0\\0&0&0&0&0&0&1&0&0\\0&0&0&0&0&1&0&0&0\\0&0&0&0&1&0&0&0&0\\0&0&0&1&0&0&0&0&0\\
0&0&1&0&0&0&0&0&0\\0&1&0&0&0&0&0&0&0\\1&0&0&0&0&0&0&0&0\end{array}\right]$$
The matrix you seek is then:
$$(\textrm{vec}^{-1}({\bf R}\textrm{vec}({\bf A})))^T = \left[\begin{array}{ccc}9&6&3\\8&5&2\\7&4&1\end{array}\right]$$
Which is the transpose of the special reflection $\bf R$ of the vectorization.
In some sense a reflected transpose.
We can of course do all operations inside the vectorization with $\bf T$ transpose:
$${\bf T} = \left[\begin{array}{ccccccccc}1&0&0&0&0&0&0&0&0\\0&0&0&1&0&0&0&0&0\\0&0&0&0&0&0&1&0&0\\0&1&0&0&0&0&0&0&0\\0&0&0&0&1&0&0&0&0\\0&0&0&0&0&0&0&1&0\\0&0&1&0&0&0&0&0&0\\0&0&0&0&0&1&0&0&0\\0&0&0&0&0&0&0&0&1\end{array}\right]$$
And we get:
$$\textrm{vec}^{-1}({\bf TR}\textrm{vec}({\bf A})) = \left[\begin{array}{ccc}9&6&3\\8&5&2\\7&4&1\end{array}\right]$$
If $v$ is in the span of the set you mentioned, then it is expressible as a linear combination of vectors in the original set. This linear combination solves the equation you say is "inconsistent".
If what you mean by "inconsistent" is that a matrix made of the $v_i$ has zero determinant, this simply expresses the fact that they are linearly dependent, and that they will span a space of fewer than $3$ dimensions. But the span still exists. If $v_1\neq 0$ then $v_1$ is definitely in the span, for example.
Best Answer
Note that
$$(\vec x \cdot \vec x)^2=|\vec x|^4=(x_1^2+x_2^2)^2$$
Thus: $$\frac{d(\vec x \cdot \vec x)^2}{d\vec x} =\left(\frac{\partial(\vec x \cdot \vec x)^2}{\partial x_1},\frac{\partial(\vec x \cdot \vec x)^2}{\partial x_2}\right) =(4x_1(x_1^2+x_2^2),4x_2(x_1^2+x_2^2))$$
$$\left|\frac{d(\vec x \cdot \vec x)^2}{d\vec x} \right|=\sqrt{16x_1^2(x_1^2+x_2^2)^2+16x_2^2(x_1^2+x_2^2)^2}=4(x_1^2+x_2^2)^{\frac32}=4|\vec x|^3$$
You can also apply the ordinary rules using the product rule:
$$\frac{d(\vec x \cdot \vec x)^2}{d\vec x} =2(\vec x \cdot \vec x)\vec x+2(\vec x \cdot \vec x)\vec x=4\vec x|\vec x|^2$$