Your guess as to what went wrong is correct: you’ve computed the transformation that maps the basic triangle onto the arbitrary one—the inverse of the map that’s needed.† Try plugging in any of the vertices into your formula to see this: you won’t get $(0,0)$, $(1,0)$ or $(1,1)$. The transformation in the paper goes in the correct direction. In fact, it’s precisely the inverse of yours.
To get from your solution to the correct one, solve the two equations that it represents for $x$ and $y$. (Their roles are reversed in the two formulas.) Alternatively, start from scratch with the correct direction:
From $(s1,t1)\to(0,0)$ you get $$s_1 a_{11}+t_1 a_{12}+a_{31} = 0 \\ s_1 a_{21} + t_1 a_{22} + a_{32} = 0$$ and so on for the other three point pairs. Solve the resulting system of 6 equations as you did before.
As for the $J$ in that formula, it’s the determinant of the matrix in your formula, which appears naturally when that matrix is inverted. The $a$’s in the definition of $J$ are meant to be elements of a generic $2\times2$ matrix. In this particular instance, they’re the elements of the matrix in your formula, i.e., $a_{11}=s_2-s_1$ and so on, giving $$J = (s_2-s_1)(t_3-t_2-t_1)-(t_2-t_1)(s_3-s_2-s_1).$$
If you’re familiar with matrices, it’s fairly easy to invert your formula directly. That transformation consists of a linear transformation followed by a translation, i.e., it has the form $\mathbf p = M\mathbf x+\mathbf p_1$, so to invert it you undo each of those components in reverse order: $\mathbf x = M^{-1}(\mathbf p-\mathbf p_1) = M^{-1}\mathbf p-M^{-1}\mathbf p_1$ Applying this to your transformation after swapping $(x,y)$ and $(s,t)$, we translate back: $$\begin{bmatrix}x\\y\end{bmatrix} - \begin{bmatrix}s_1\\t_1\end{bmatrix} = \begin{bmatrix}s_2-s_1&s_3-s_2\\t_2-t_1&t_3-t_2\end{bmatrix}\begin{bmatrix}s\\t\end{bmatrix}$$ then invert the linear part of the transformation $$\begin{bmatrix}s_2-s_1&s_3-s_2\\t_2-t_1&t_3-t_2\end{bmatrix}^{-1} \left( \begin{bmatrix}x\\y\end{bmatrix} - \begin{bmatrix}s_1\\t_1\end{bmatrix} \right) = \begin{bmatrix}s\\t\end{bmatrix}.$$ Now invert the matrix and distribute the multiplication: $$\begin{bmatrix}s\\t\end{bmatrix} = \frac1J\begin{bmatrix} (t_3-t_2) & (s_2-s_3) \\ (t_1-t_2) & (s_2-s_1) \end{bmatrix} \begin{bmatrix}x\\y\end{bmatrix} - \frac1J\begin{bmatrix} (t_3-t_2)s_1 + (s_2-s_3)t_1 \\ (t_1-t_2)s_1 + (s_2-s_1)t_1 \end{bmatrix},$$ with $J$ as above.
If you use homogeneous coordinates, then there’s a conceptually simple way to construct the required transformation. Recalling that the columns of a transformation matrix are the images of the basis vectors, we can see that the matrix $$\begin{bmatrix} x_1 & x_2 & x_3 \\ y_1 & y_2 & y_3 \\ 1&1&1\end{bmatrix}$$ maps the standard basis onto the points $(x_1,y_1)$, $(x_2,y_2)$ and $(x_3,y_3)$. So, to map the given triangle onto the basic one, we can first map its vertices onto the standard basis, then map that onto the standard triangle. The resulting transformation is given by the matrix product $$\begin{bmatrix}0&1&1\\0&0&1\\1&1&1\end{bmatrix} \begin{bmatrix}s_1&s_2&s_3\\t_1&t_2&t_3\\1&1&1\end{bmatrix}^{-1}.$$ If you do this correctly, the last row of the resulting matrix will be $[0,0,1]$, which tells you that you have an affine transformation as required. You can use this same method to construct the affine transformation between any two triangles: just put the appropriate destination vertex coordinates into the left-hand matrix.
† Actually, there’s a mistake somewhere in your calculations because your transformation doesn’t map $(1,1)$ to $(s_3,t_3)$, but that’s not the important thing here.
Best Answer
Suppose the three lines are given by
$ p(t_1) = p_0 + t_1 v_1 $
$ q(t_2) = q_0 + t_2 v_2 $
$ r(t_3) = r_0 + t_3 v_3 $
and that there exists constants $\alpha$ and $\beta$ such that
$ v_3 = \alpha v_1 + \beta v_2 $
Then define $ v_4 = v_1 \times v_2 $, then $v_4$ will be perpendicular to $v_1$ and to $v_2$ and also to $v_3$. Thus we now have the line
$ s(t_4) = s_0 + t_4 v_4 $
We want $s(t_4)$ to intersect $p(t_1) , q(t_2) , r(t_3) $ at three different values of $t_4$, let's call them $t_4, t_5, t_6$, so we now have
$ s_0 + t_4 v_4 = p_0 + t_1 v_1 $
$ s_0 + t_5 v_4 = q_0 + t_2 v_2 $
$ s_0 + t_6 v_4 = r_0 + t_3 v_3 $
From the first equation, we have
$ s_0 = p_0 + t_1 v_1 - t_4 v_4 $
Substituting in the other two equations
$ p_0 + t_1 v_1 + (t_5 - t_4 ) v_4 = q_0 + t_2 v_2 $
$ p_0 + t_1 v_1 + (t_6 - t_4) v_4 = r_0 + t_3 v_3 $
This gives the linear system
$ A t = b $
where
$A = \begin{bmatrix} v_1 && - v_2 && 0 && - v_4 && v_4 && 0 \\ v_1 && 0 && - v_3 && - v_4 && 0 && v_4 \end{bmatrix} $
and
$ b = \begin{bmatrix} q_0 - p_0 \\ r_0 - p_0 \end{bmatrix} $
And
$ t = [t_1, t_2, t_3, t_4, t_5, t_6 ]^T $
Note that $A$ is a $6 \times 6$ matrix.
If this system has a solution, then it is possible to pass a fourth line through the three given lines, with it being perpendicular to them.