General Idea
You were on the right track with those matrices! Instead of trying to solve the system of equations in your $M$ matrix, it's easier to figure out each transformation individually and then just multiply the results together.
Let's start with a (slightly) simplified version, where the initial line segment lies along the $x$-axis, with one point at the origin. With that assumption, it becomes easier to visualize how we could change one segment into the other:
- Scale it so the lengths match.
- Rotate it (about the origin) so that it's at the same angle as the other segment.
- Translate it to the right place.
(Your $M$ matrix basically does this- although it scales, translates, then rotates, which won't end up with the right result since you're rotating about the origin.)
Scale
The scale factor is just the ratio of the two lengths ($\frac{l_2}{l_1}$), since you want the final length to be $l_2$ and $l_1\times\frac{l_2}{l_1}=l_2$. To scale points equally along both axes, you use the scaling matrix you had with the scaling factor:
$$\left(\begin{array}{ccc}
\frac{l_2}{l_1} & 0 & 0\\
0 & \frac{l_2}{l_1} & 0\\
0 & 0 & 1
\end{array}\right)$$
To calculate $l_1$ and $l_2$, we can use the distance formula ($\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}$).
Rotate
Because of our simplifying assumption that the initial segment starts at $(0,0)$ and lies along the $x$-axis, figuring out the rotation matrix is just a matter of figuring out the angle the second segment makes with the $x$-axis. To do this, we can use atan2, which is just like $tan^{-1}$ but takes the signs/quadrants into account.
$atan2$ works with vectors, so we need to shift the second segment to the origin before doing the calculation. This is done by subtracting the coordinates of one point of the segment from the other.
For example, for a segment with points $a=(a_x,a_y)$ and $b=(b_x,b_y)$, the angle it makes with the $x$-axis is $\theta=\text{atan2}(b_y-a_y, b_x-a_x)$, which can be plugged into your transformation matrix:
$$\left(\begin{array}{ccc}
\cos\theta & -\sin\theta & 0\\
\sin\theta & \cos\theta & 0\\
0 & 0 & 1
\end{array}\right)$$
Translate
At this point, the translation is easy, we just need to shift $(0,0)$ to the corresponding point on the final segment:
$$\left(\begin{array}{ccc}
1 & 0 & x_3\\
0 & 1 & y_3\\
0 & 0 & 1
\end{array}\right)$$
Putting Them Together
The final transformation matrix is just the three we've derived combined:
$$
M=
\left(\begin{array}{ccc}
1 & 0 & x_3\\
0 & 1 & y_3\\
0 & 0 & 1
\end{array}\right)
\left(\begin{array}{ccc}
\cos\theta & -\sin\theta & 0\\
\sin\theta & \cos\theta & 0\\
0 & 0 & 1
\end{array}\right)
\left(\begin{array}{ccc}
\frac{l_2}{l_1} & 0 & 0\\
0 & \frac{l_2}{l_1} & 0\\
0 & 0 & 1
\end{array}\right)
$$
But... What If The First Segment Isn't In A Nice Place?
We can put it there! Using the same techniques we've already used, all you have to do is:
- Translate the first segment so that one point is at the origin.
- Rotate it so that it lies along the $x$-axis.
Hint: Write the vector $(6,7,0)^{T}$ as a linear combination of $(1,2,1)^{T}$, $(2,2,0)^{T}$, and $(3,0,6)^{T}$. Then use the formulas from the first line.
This will give an equation like $A(c_{1}v_{1}+c_{2}v_{2}+c_{3}v_{3}) = c_{1}Av_{1}+c_{2}Av_{2}+c_{3}Av_{3}$. Your formulas at the beginning will give a nice way to write $Av_{i}$ (for example, $Av_{1}=v_{1}$), and should give a clue on how to get an answer for general $n$. You can take the limit from here.
Best Answer
The computation is correct, and we cannot expect that $P$ is unique in general. A somewhat extreme case is $A=B=0$, where any invertible $P$ will do.