The transformation you're looking for has this form:
$$\left[
\begin{array}{ccc}
t_1 & t_2 & t_3\\
t_4 & t_5 & t_6\\
0 & 0 & 1\\
\end{array}
\right]
\left[
\begin{array}{ccc}
a_1 & b_1 & c_1\\
a_2 & b_2 & c_2 \\
1 & 1 & 1 \\
\end{array}
\right]
= \left[\begin{array}{ccc}
x_1 & y_1 & z_1\\
x_2 & y_2 & z_2\\
1 & 1 & 1
\end{array}
\right]
$$
or
$${\bf T A} = {\bf X}$$
so
$${\bf T} = {\bf X}{\bf A}^{-1}$$
Now $\bf T$ is a transformation matrix you can use on any point, like
$${\bf T}\left[\begin{array}{c}a_1\\a_2\\1\end{array}\right] = \left[\begin{array}{c}x_1\\x_2\\1\end{array}\right]$$
It's linear, so it has the halfway point property you were looking for.
$$
{\bf A} = \left[\begin{array}{ccc}
-3 & 0 & 3\\
0 & 3 & 0\\
1 & 1 & 1\\
\end{array}\right]
$$
$$
\left[\begin{array}{ccc|ccc}
-3 & 0 & 3 & 1 & 0 & 0\\
0 & 3 & 0 & 0 & 1 & 0\\
1 & 1 & 1 & 0 & 0 & 1\\
\end{array}\right]
$$
$$
\left[\begin{array}{ccc|ccc}
1 & 0 & -1 & -\frac13 & 0 & 0\\
0 & 3 & 0 & 0 & 1 & 0\\
0 & 1 & 2 & \frac13 & 0 & 1\\
\end{array}\right]
$$
$$
\left[\begin{array}{ccc|ccc}
1 & 0 & -1 & -\frac13 & 0 & 0\\
0 & 1 & 0 & 0 & \frac13 & 0\\
0 & 0 & 2 & \frac13 & -\frac13 & 1\\
\end{array}\right]
$$
$$
\left[\begin{array}{ccc|ccc}
1 & 0 & 0 & -\frac16 & -\frac16 & \frac12\\
0 & 1 & 0 & 0 & \frac13 & 0\\
0 & 0 & 1 & \frac16 & -\frac16 & \frac12\\
\end{array}\right]\\
\text{so } {\bf A}^{-1} =
\left[\begin{array}{ccc}
-\frac16 & -\frac16 & \frac12\\
0 & \frac13 & 0\\
\frac16 & -\frac16 & \frac12\\
\end{array}\right]$$
$${\bf XA}^{-1} =
\left[\begin{array}{ccc}
2 & 3 & 4\\
3 & 2 & 3\\
1 & 1 & 1\\
\end{array}\right]
\left[\begin{array}{ccc}
-\frac16 & -\frac16 & \frac12\\
0 & \frac13 & 0\\
\frac16 & -\frac16 & \frac12\\
\end{array}\right]
=
\left[\begin{array}{ccc}
\frac13 & 0 & 3\\
0 & -\frac13 & 3\\
0 & 0 & 1\\
\end{array}\right]
$$
General Idea
You were on the right track with those matrices! Instead of trying to solve the system of equations in your $M$ matrix, it's easier to figure out each transformation individually and then just multiply the results together.
Let's start with a (slightly) simplified version, where the initial line segment lies along the $x$-axis, with one point at the origin. With that assumption, it becomes easier to visualize how we could change one segment into the other:
- Scale it so the lengths match.
- Rotate it (about the origin) so that it's at the same angle as the other segment.
- Translate it to the right place.
(Your $M$ matrix basically does this- although it scales, translates, then rotates, which won't end up with the right result since you're rotating about the origin.)
Scale
The scale factor is just the ratio of the two lengths ($\frac{l_2}{l_1}$), since you want the final length to be $l_2$ and $l_1\times\frac{l_2}{l_1}=l_2$. To scale points equally along both axes, you use the scaling matrix you had with the scaling factor:
$$\left(\begin{array}{ccc}
\frac{l_2}{l_1} & 0 & 0\\
0 & \frac{l_2}{l_1} & 0\\
0 & 0 & 1
\end{array}\right)$$
To calculate $l_1$ and $l_2$, we can use the distance formula ($\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}$).
Rotate
Because of our simplifying assumption that the initial segment starts at $(0,0)$ and lies along the $x$-axis, figuring out the rotation matrix is just a matter of figuring out the angle the second segment makes with the $x$-axis. To do this, we can use atan2, which is just like $tan^{-1}$ but takes the signs/quadrants into account.
$atan2$ works with vectors, so we need to shift the second segment to the origin before doing the calculation. This is done by subtracting the coordinates of one point of the segment from the other.
For example, for a segment with points $a=(a_x,a_y)$ and $b=(b_x,b_y)$, the angle it makes with the $x$-axis is $\theta=\text{atan2}(b_y-a_y, b_x-a_x)$, which can be plugged into your transformation matrix:
$$\left(\begin{array}{ccc}
\cos\theta & -\sin\theta & 0\\
\sin\theta & \cos\theta & 0\\
0 & 0 & 1
\end{array}\right)$$
Translate
At this point, the translation is easy, we just need to shift $(0,0)$ to the corresponding point on the final segment:
$$\left(\begin{array}{ccc}
1 & 0 & x_3\\
0 & 1 & y_3\\
0 & 0 & 1
\end{array}\right)$$
Putting Them Together
The final transformation matrix is just the three we've derived combined:
$$
M=
\left(\begin{array}{ccc}
1 & 0 & x_3\\
0 & 1 & y_3\\
0 & 0 & 1
\end{array}\right)
\left(\begin{array}{ccc}
\cos\theta & -\sin\theta & 0\\
\sin\theta & \cos\theta & 0\\
0 & 0 & 1
\end{array}\right)
\left(\begin{array}{ccc}
\frac{l_2}{l_1} & 0 & 0\\
0 & \frac{l_2}{l_1} & 0\\
0 & 0 & 1
\end{array}\right)
$$
But... What If The First Segment Isn't In A Nice Place?
We can put it there! Using the same techniques we've already used, all you have to do is:
- Translate the first segment so that one point is at the origin.
- Rotate it so that it lies along the $x$-axis.
Best Answer
You wrote:
but this is mistaken; I think it should be
$\vec{f}(\overline{p_2p_3})=\overline{f(p_\color{red}2)f(p_\color{red}3)}=\overline{p_3p_0}=-\overline{p_0p_3}=-(\overline{p_0p_1}+\overline{p_1p_2}+\overline{p_2p_3})$
You ask "how do I find the $f(p_0)$ coordinates?"
I'm not sure exactly how your text/teacher is expressing affine xforms as matrices, so I don't know how to fill in the first column, but you do know that $f(p_0) = p_1$, so you can write $$ f(p_0) = p_1 = p_0 + (p_1 - p_0) = p_0 + b_1 $$ where $b_1$ is the first element of your basis. That makes me suspect that your matrix should be something like $$ \left(\begin{array}{cccc} 1&0&0&0\\ \color{red}1&0&0&-1\\ \color{red}0&1&0&-1\\ \color{red}0&0&1&-1\\ \end{array}\right) $$