Does this mean the system is inconsistent?
No. My response would be why do you think $D \mathbf x = \mathbf b$ would hold for the same $\mathbf x$ that makes $A\mathbf x=\mathbf b$ true?
There are really two systems here: $A\mathbf x = \mathbf b$ and $D\mathbf x = \mathbf b$. The consistency of one has nothing to do with the consistency of the other.
A system is consistent if and only if it has at least one solution. So if you found a solution for the system $A\mathbf x = \mathbf b$, then the system is consistent. And you did find the right solution. Now turn your attention towards $D \mathbf x = \mathbf b$, keeping it completely separate from $A\mathbf x = \mathbf b$.
Your guess as to what went wrong is correct: you’ve computed the transformation that maps the basic triangle onto the arbitrary one—the inverse of the map that’s needed.† Try plugging in any of the vertices into your formula to see this: you won’t get $(0,0)$, $(1,0)$ or $(1,1)$. The transformation in the paper goes in the correct direction. In fact, it’s precisely the inverse of yours.
To get from your solution to the correct one, solve the two equations that it represents for $x$ and $y$. (Their roles are reversed in the two formulas.) Alternatively, start from scratch with the correct direction:
From $(s1,t1)\to(0,0)$ you get $$s_1 a_{11}+t_1 a_{12}+a_{31} = 0 \\ s_1 a_{21} + t_1 a_{22} + a_{32} = 0$$ and so on for the other three point pairs. Solve the resulting system of 6 equations as you did before.
As for the $J$ in that formula, it’s the determinant of the matrix in your formula, which appears naturally when that matrix is inverted. The $a$’s in the definition of $J$ are meant to be elements of a generic $2\times2$ matrix. In this particular instance, they’re the elements of the matrix in your formula, i.e., $a_{11}=s_2-s_1$ and so on, giving $$J = (s_2-s_1)(t_3-t_2-t_1)-(t_2-t_1)(s_3-s_2-s_1).$$
If you’re familiar with matrices, it’s fairly easy to invert your formula directly. That transformation consists of a linear transformation followed by a translation, i.e., it has the form $\mathbf p = M\mathbf x+\mathbf p_1$, so to invert it you undo each of those components in reverse order: $\mathbf x = M^{-1}(\mathbf p-\mathbf p_1) = M^{-1}\mathbf p-M^{-1}\mathbf p_1$ Applying this to your transformation after swapping $(x,y)$ and $(s,t)$, we translate back: $$\begin{bmatrix}x\\y\end{bmatrix} - \begin{bmatrix}s_1\\t_1\end{bmatrix} = \begin{bmatrix}s_2-s_1&s_3-s_2\\t_2-t_1&t_3-t_2\end{bmatrix}\begin{bmatrix}s\\t\end{bmatrix}$$ then invert the linear part of the transformation $$\begin{bmatrix}s_2-s_1&s_3-s_2\\t_2-t_1&t_3-t_2\end{bmatrix}^{-1} \left( \begin{bmatrix}x\\y\end{bmatrix} - \begin{bmatrix}s_1\\t_1\end{bmatrix} \right) = \begin{bmatrix}s\\t\end{bmatrix}.$$ Now invert the matrix and distribute the multiplication: $$\begin{bmatrix}s\\t\end{bmatrix} = \frac1J\begin{bmatrix} (t_3-t_2) & (s_2-s_3) \\ (t_1-t_2) & (s_2-s_1) \end{bmatrix} \begin{bmatrix}x\\y\end{bmatrix} - \frac1J\begin{bmatrix} (t_3-t_2)s_1 + (s_2-s_3)t_1 \\ (t_1-t_2)s_1 + (s_2-s_1)t_1 \end{bmatrix},$$ with $J$ as above.
If you use homogeneous coordinates, then there’s a conceptually simple way to construct the required transformation. Recalling that the columns of a transformation matrix are the images of the basis vectors, we can see that the matrix $$\begin{bmatrix} x_1 & x_2 & x_3 \\ y_1 & y_2 & y_3 \\ 1&1&1\end{bmatrix}$$ maps the standard basis onto the points $(x_1,y_1)$, $(x_2,y_2)$ and $(x_3,y_3)$. So, to map the given triangle onto the basic one, we can first map its vertices onto the standard basis, then map that onto the standard triangle. The resulting transformation is given by the matrix product $$\begin{bmatrix}0&1&1\\0&0&1\\1&1&1\end{bmatrix} \begin{bmatrix}s_1&s_2&s_3\\t_1&t_2&t_3\\1&1&1\end{bmatrix}^{-1}.$$ If you do this correctly, the last row of the resulting matrix will be $[0,0,1]$, which tells you that you have an affine transformation as required. You can use this same method to construct the affine transformation between any two triangles: just put the appropriate destination vertex coordinates into the left-hand matrix.
† Actually, there’s a mistake somewhere in your calculations because your transformation doesn’t map $(1,1)$ to $(s_3,t_3)$, but that’s not the important thing here.
Best Answer
If $v$ is in the span of the set you mentioned, then it is expressible as a linear combination of vectors in the original set. This linear combination solves the equation you say is "inconsistent".
If what you mean by "inconsistent" is that a matrix made of the $v_i$ has zero determinant, this simply expresses the fact that they are linearly dependent, and that they will span a space of fewer than $3$ dimensions. But the span still exists. If $v_1\neq 0$ then $v_1$ is definitely in the span, for example.