We need to find a transformation that maps the segments. It is sufficient to restrict the search to affine maps and to map only the end points
$$
z_1=\left[\matrix{x_1\\y_1}\right]\ \mapsto w_1=\left[\matrix{x_3\\y_3}\right],\quad\text{and}\quad
z_2=\left[\matrix{x_2\\y_2}\right]\ \mapsto \ w_2=\left[\matrix{x_4\\y_4}\right].
$$
Case I: $\text{rank}\,[z_1\ z_2]=2$. Then the linear transformation
$$
w=Az\qquad\text{where}\qquad A=[w_1\ w_2]\cdot [z_1\ z_2]^{-1}
$$
does the job.
Case II: $z_1\parallel z_2$, but $z_1\ne z_2$. Then we need to consider an affine map $w=Az+z_0$ to shift the $w$-vectors. Let $\lambda z_1+\mu z_2=0$. Then
$$
\left\{
\begin{array}{lcl}
\lambda Az_1+\lambda z_0&=&\lambda w_1,\\
\mu Az_2+\mu z_0&=&\mu w_2,\\
\end{array}
\right.\quad\Rightarrow\quad \underbrace{A(\lambda z_1+\mu z_2)}_{=0}+(\lambda+\mu)z_0=\lambda w_1+\mu w_2.
$$
It gives $z_0=\frac{\lambda w_1+\mu w_2}{\lambda+\mu}$. Note that $\lambda+\mu\ne 0$ by the assumption that $z_1\ne z_2$. With this $z_0$ any $A$ that maps only one end point, i.e. $Az_1=w_1-z_0$ would suffice (not unique).
Case III: $z_1=z_2$ is trivial. If $w_1=w_2$ (point-to-point) then any $A$ with $Az_1=w_1$ works. If $w_1\ne w_2$ (point-to-segment) then no such transformation (even non-linear) exists.
I think you mix everything. When we speak about congruence, the matrices do not act on vectors and, consequently, the angles cannot be preserved. The "similar" in "similar triangles" has nothing to do with the "similar" of "similar matrices". When you write "affine transformations (i.e., uniform scaling, rotation, reflection, translation[?])", you are far from the definition of an affine function.
An affine function is in the form $f:x\in\mathbb{R}^n\rightarrow Ax+b\in\mathbb{R}^p$ where $A\in M_{p,n},b\in\mathbb{R}^p$ are fixed.
Similarity of triangles in the plane. It is associated (up to a translation) to a transformation in the form $z\in\mathbb{C}\rightarrow az$ (for direct similarity) or $z\in\mathbb{C}\rightarrow a\overline{z}$ (for inverse similarity) where $a=u+iv$ is a fixed complex. It's a composition of homothety, rotation and, eventually, symmetry. The associated linear application has the form $\begin{pmatrix}u&-v\\v&u\end{pmatrix}$ (in a vector subspace of $M_2$ that is isomorphic to $\mathbb{C}$).
When we speak about similarity of matrices, the matrix $A$ is considered as a linear function and acts on vectors:
$y=Ax$. By a change of basis $y=Py',x=Px'$ and $y'=P^{-1}APx$, that is, the new matrix is $P^{-1}AP$.
When we speak about congruence of matrices, the matrix $A$ is considered as a bilinear form and acts on a couple of vectors $(x,y)$: $\phi(x,y)=x^TAy$. By a change of basis $y=Py',x=Px'$ and $x^TAy=x'^TP^TAPy'$, that is, the new matrix is $P^TAP$.
Note that, if we use the standard inner product $<.>$, $\phi(x,y)=<Ay,x>=<y,A^Tx>$, that is the definition of the adjoint $A^T$.
- The orthogonal matrices $U$ bridge the two previous notions because $U^T=U^{-1}$: by an orthonormal change of basis, a matrix $A$ can be considered as the matrix of a bilinear form or as the matrix of a linear function.
EDIT. Answer to jjjjjj . Let $T_1,T_2$ be two triangles. According to the standard definitions,
$T_1,T_2$ are similar iff $f(T_1)=T_2$ where $f $ is affine (cf. 1.) with $A=\lambda U$ where $\lambda>0$ and $U$ is orthogonal.
$T_1,T_2$ are congruent iff $f(T_1)=T_2$ where $f $ is affine (cf. 1.) with $A=U$ orthogonal.
Note that if $\det(U)=1$, then the angles are preserved and if $\det(U)=-1$, then the angles are transformed in their opposite.
Assume that $T_1,T_2$ are congruent and placed on a sheet. If $\det(U)=1$, then we can drag $T_1$ onto $T_2$. If $\det(U)=-1$, then we flip $T_1$, after that, we drag it onto $T_2$.
Best Answer
A non-invertible transformation collapses the space along some direction(s). If you know about eigenvalues and eigenvectors, the eigenvectors of $0$ give you the directions in which this collapse happens.
Rotations are isometries, so they never contribute to making a transformation non-invertible. A shear shouldn’t cause any collapse, either. It shifts things parallel to some direction, which keeps areas/volumes constant, so there’s no collapse there, either. That leaves scaling as the likely culprit, but that’s easy to detect: the result is not invertible iff any of the scale factors is zero.