The key fact here is that $u,v,w$ are affinely independent. Then any point in the plane can be expressed as a unique affine combination of $u,v,w$.
You have $r = \frac{2}{3} u + {1 \over 3} v $, $p = {2 \over 3} r + {1 \over 3 } w$. Combining gives $p = {4 \over 9} u + {2 \over 9} v + {1 \over 3} w$.
We have $q = tp+(1-t) v$ for some $t$, and $t$ is such that the multiplier of $v$ is zero.
So, $q = {4 \over 9}t u + ({2 \over 9}t + (1-t)) v + {1 \over 3}t w$. Setting the multiplier of $v$ to zero gives $t = {9 \over 7}$, and so we have $q = {4 \over 7} u + {3 \over 7} w$.
Your guess as to what went wrong is correct: you’ve computed the transformation that maps the basic triangle onto the arbitrary one—the inverse of the map that’s needed.† Try plugging in any of the vertices into your formula to see this: you won’t get $(0,0)$, $(1,0)$ or $(1,1)$. The transformation in the paper goes in the correct direction. In fact, it’s precisely the inverse of yours.
To get from your solution to the correct one, solve the two equations that it represents for $x$ and $y$. (Their roles are reversed in the two formulas.) Alternatively, start from scratch with the correct direction:
From $(s1,t1)\to(0,0)$ you get $$s_1 a_{11}+t_1 a_{12}+a_{31} = 0 \\ s_1 a_{21} + t_1 a_{22} + a_{32} = 0$$ and so on for the other three point pairs. Solve the resulting system of 6 equations as you did before.
As for the $J$ in that formula, it’s the determinant of the matrix in your formula, which appears naturally when that matrix is inverted. The $a$’s in the definition of $J$ are meant to be elements of a generic $2\times2$ matrix. In this particular instance, they’re the elements of the matrix in your formula, i.e., $a_{11}=s_2-s_1$ and so on, giving $$J = (s_2-s_1)(t_3-t_2-t_1)-(t_2-t_1)(s_3-s_2-s_1).$$
If you’re familiar with matrices, it’s fairly easy to invert your formula directly. That transformation consists of a linear transformation followed by a translation, i.e., it has the form $\mathbf p = M\mathbf x+\mathbf p_1$, so to invert it you undo each of those components in reverse order: $\mathbf x = M^{-1}(\mathbf p-\mathbf p_1) = M^{-1}\mathbf p-M^{-1}\mathbf p_1$ Applying this to your transformation after swapping $(x,y)$ and $(s,t)$, we translate back: $$\begin{bmatrix}x\\y\end{bmatrix} - \begin{bmatrix}s_1\\t_1\end{bmatrix} = \begin{bmatrix}s_2-s_1&s_3-s_2\\t_2-t_1&t_3-t_2\end{bmatrix}\begin{bmatrix}s\\t\end{bmatrix}$$ then invert the linear part of the transformation $$\begin{bmatrix}s_2-s_1&s_3-s_2\\t_2-t_1&t_3-t_2\end{bmatrix}^{-1} \left( \begin{bmatrix}x\\y\end{bmatrix} - \begin{bmatrix}s_1\\t_1\end{bmatrix} \right) = \begin{bmatrix}s\\t\end{bmatrix}.$$ Now invert the matrix and distribute the multiplication: $$\begin{bmatrix}s\\t\end{bmatrix} = \frac1J\begin{bmatrix} (t_3-t_2) & (s_2-s_3) \\ (t_1-t_2) & (s_2-s_1) \end{bmatrix} \begin{bmatrix}x\\y\end{bmatrix} - \frac1J\begin{bmatrix} (t_3-t_2)s_1 + (s_2-s_3)t_1 \\ (t_1-t_2)s_1 + (s_2-s_1)t_1 \end{bmatrix},$$ with $J$ as above.
If you use homogeneous coordinates, then there’s a conceptually simple way to construct the required transformation. Recalling that the columns of a transformation matrix are the images of the basis vectors, we can see that the matrix $$\begin{bmatrix} x_1 & x_2 & x_3 \\ y_1 & y_2 & y_3 \\ 1&1&1\end{bmatrix}$$ maps the standard basis onto the points $(x_1,y_1)$, $(x_2,y_2)$ and $(x_3,y_3)$. So, to map the given triangle onto the basic one, we can first map its vertices onto the standard basis, then map that onto the standard triangle. The resulting transformation is given by the matrix product $$\begin{bmatrix}0&1&1\\0&0&1\\1&1&1\end{bmatrix} \begin{bmatrix}s_1&s_2&s_3\\t_1&t_2&t_3\\1&1&1\end{bmatrix}^{-1}.$$ If you do this correctly, the last row of the resulting matrix will be $[0,0,1]$, which tells you that you have an affine transformation as required. You can use this same method to construct the affine transformation between any two triangles: just put the appropriate destination vertex coordinates into the left-hand matrix.
† Actually, there’s a mistake somewhere in your calculations because your transformation doesn’t map $(1,1)$ to $(s_3,t_3)$, but that’s not the important thing here.
Best Answer
Since you’re being asked to represent an affine transformation as a single matrix, I presume that you’re familiar with homogeneous coordinates. So, you’re looking for a matrix of the form $$M=\begin{bmatrix}a_{11}&a_{12}&t_{1} \\ a_{21} & a_{22} & t_2 \\ 0&0&1\end{bmatrix}$$ such that $$M\left(\frac12,0,1\right)^T = \left(\frac12,1,1\right)^T \\ M\left(\frac12,\frac12,1\right)T = (0,0,1)^T \\ M(1,0,1)^T = (1,0,1)^T.$$ If you multiply these out, you will have a system of linear equations in the unknown entries of $M$ to solve.
On the other hand, if you package the three conditions up as a single matrix equation, you can derive an expression for $M$ as the product of two matrices. We have $$M \begin{bmatrix}\frac12&\frac12&1 \\ 0&\frac12&0 \\ 1&1&1\end{bmatrix} = \begin{bmatrix} \frac12&0&1 \\ 1&0&0 \\ 1&1&1 \end{bmatrix}$$ and since the matrix on the left-hand side is nonsingular (the triangle isn’t degenerate, so the homogeneous coordinates of its vertices are linearly independent), $$M = \begin{bmatrix}\frac12&0&1\\1&0&0\\1&1&1\end{bmatrix} \begin{bmatrix}\frac12&\frac12&1\\0&\frac12&0\\1&1&1\end{bmatrix}^{-1}.$$ Recall that the columns of a transformation matrix are the images of the basis vectors. Geometrically, then, the right-hand matrix maps the vertices of the first triangle to $(1,0)$, $(0,1)$ and $(0,0)$, respectively, and the left-hand matrix maps these points to the second triangle’s vertices.
Using your observation that the two triangles are isosceles and equal sides of each one are paired, you might instead try to compose the transformation out of more basic ones. There’s probably going to be a rotation and a scaling, together with a translation or two so that everything lines up correctly. Note, however, that there’s also an orientation change: the first triangle’s vertices are traversed clockwise, but the second one’s are traversed counterclockwise, so you’ll need to throw in a reflection as well. Here’s one possible sequence:
You can either multiply the five matrices that represent these simpler transformations together, or work in coordinates, applying the formula for each transformation in turn, then extracting the elements of the composite matrix after simplifying the resulting expressions.