Let's say $\epsilon$ is the line defined by A and B and $K_1,K_2$ and $K_3$ the centers of the given circles.
Then we have the following 3 equations that will help us define $x_3,y_3,r_3$
$1$. $\sqrt{(x_3-x_2)^2+(y_3-y_2)^2}=r_3+r_2$ or $\sqrt{(x_3-x_2)^2+(y_3-y_2)^2}=|r_3-r_2|$
That is, the distance of the centers $K_2,K_3$ equals the sum of the radius $r_3,r_2$ if the circles are tangent outwardly or the $|r_3-r_2|$ if they're tangent inwardly.
$2$. $\sqrt{(x_3-x_1)^2+(y_3-y_1)^2}=r_3+r_1$ or $\sqrt{(x_3-x_1)^2+(y_3-y_1)^2}=|r_3-r_1|$
$3$. The distance of the $K_3$ from $\epsilon$ equals $r_3$ if you write $\epsilon :Ax+By+C=0$ then $r_3=\frac{|Ax_3+By_3+c|}{\sqrt{A^2+B^2}}$
Let's use
$$\vec{p}_0 = \left[\begin{matrix} x_A \\ y_A \end{matrix}\right],
\vec{q}_0 = \left[\begin{matrix} x_{A^\prime} \\ y_{A^\prime} \end{matrix}\right]$$
for the endpoints of the line segment in the first coordinate system, and
$$\vec{p}_1 = \left[\begin{matrix} x_B \\ y_B \end{matrix}\right],
\vec{q}_1 = \left[\begin{matrix} x_{B^\prime} \\ y_{B^\prime} \end{matrix}\right]$$
the corresponding points in the second coordinate system.
Let's define two scalars, corresponding to the lengths of the line segments, and two unit vectors, corresponding to the directions of the line segments:
$$\begin{aligned}
L_0 &= \left\lVert \vec{q}_0 - \vec{p}_0 \right\rVert \\
L_1 &= \left\lVert \vec{q}_1 - \vec{p}_1 \right\rVert \\
\hat{n}_0 &= \displaystyle \frac{\vec{q}_0 - \vec{p}_0}{L_0} = \left[\begin{matrix} x_0 \\ y_0 \end{matrix}\right] = \frac{1}{L_0} \left[\begin{matrix} x_{A^\prime} - x_A \\ y_{A^\prime} - y_A \end{matrix}\right] \\
\hat{n}_1 &= \displaystyle \frac{\vec{q}_1 - \vec{p}_1}{L_1} = \left[\begin{matrix} x_1 \\ y_1 \end{matrix}\right] = \frac{1}{L_1} \left[\begin{matrix} x_{B^\prime} - x_B \\ y_{B^\prime} - y_B \end{matrix}\right] \\
\end{aligned}$$
so that $\lVert\hat{n}_0\rVert = \lVert\hat{n}_1\rVert = 1$, and
$$\begin{aligned}
\vec{q}_0 &= \vec{p}_0 + L_0 \hat{n}_0 \\
\vec{q}_1 &= \vec{p}_1 + L_1 \hat{n}_1 \\
\end{aligned}$$
A matrix that rotates positive $x$ axis towards $\hat{n}_0$ is
$$\mathbf{R}_0 = \left[ \begin{matrix} x_0 & -y_0 \\ y_0 & x_0 \end{matrix} \right]$$
and the matrix that rotates positive $x$ axis towards $\hat{n}_1$ is
$$\mathbf{R}_1 = \left[ \begin{matrix} x_1 & -y_1 \\ y_1 & x_0 \end{matrix} \right]$$
Because $x_0^2 + y_0^2 = 1$ and $x_1^2 + y_1^2 = 1$, the above two matrices are orthogonal, and describe a pure rotation. Because they are orthogonal, their inverse is their transpose.
What we need, is a translation that moves $\vec{p}_0$ to origin, then inverts the rotation by $\mathbf{R}_0$, so that $\vec{p}_1$ will be at $[L_0, 0]^T$. Then, we apply a rotation by $\mathbf{R}_1$, scale by the ratio of the lengths of the line segments, and finally translate by $\vec{p}_1$. Using $\vec{v}_0$ for a point in the old coordinate system, and $\vec{v}_1$ for the corresponding point in the new coordinate system, the transform is described by
$$\vec{v}_1 = \vec{p}_1 + \frac{L_1}{L_0}\mathbf{R}_1 \mathbf{R}_0^T ( \vec{v}_0 - \vec{p}_0 )$$
or, grouping the fixed translation rightmost,
$$\vec{v}_1 = \left(\frac{L_1}{L_0}\mathbf{R}_1 \mathbf{R}_0^T \vec{v}_0 \right) + \left(\vec{p}_1 - \frac{L_1}{L_0} \mathbf{R}_1 \mathbf{R}_0^T \vec{p}_0 \right)$$
A typical format for expressing rotation, scaling, and translation is
$$\vec{v}_1 = \mathbf{R} \vec{v}_0 + \vec{t} \quad \iff \quad
\left[\begin{matrix} \chi_1 \\ \gamma_1 \end{matrix}\right] =
\left[\begin{matrix} u_x & -u_y \\ u_y & u_x \end{matrix}\right]
\left[\begin{matrix} \chi_0 \\ \gamma_0 \end{matrix}\right] +
\left[\begin{matrix} t_x \\ t_y \end{matrix}\right]$$
which using homogenous coordinates (which programmers often use) is
$$\left[\begin{matrix} \chi_1 \\ \gamma_1 \\ 1 \end{matrix}\right] =
\left[\begin{matrix} u_x & -u_y & t_x \\ u_y & u_x & t_y \\ 0 & 0 & 1 \end{matrix} \right] \left[ \begin{matrix} \chi_0 \\ \gamma_0 \\ 1 \end{matrix} \right]$$
where
$$\begin{aligned}
A_x &= x_{A^\prime} - x_A \\
A_y &= y_{A^\prime} - y_A \\
A^2 &= A_x^2 + A_y^2 \\
B_x &= x_{B^\prime} - x_B \\
B_y &= y_{B^\prime} - y_B \\
u_x &= \displaystyle \frac{A_x B_x + A_y B_y}{A^2} \\
u_y &= \displaystyle \frac{A_x B_y - A_y B_x}{A^2} \\
t_x &= \displaystyle x_B - \frac{x_A ( A_x B_x + A_y B_y ) + y_A ( A_y B_x - A_x B_y )}{A^2} \\
t_y &= \displaystyle y_B - \frac{x_A ( A_x B_y - A_y B_x ) + y_A ( A_x B_x + A_y B_y )}{A^2} \\
\end{aligned}$$
Best Answer
Treat everything as complex numbers, let $P$ be a point on the line segment $AA'$. The map $$P \mapsto \frac{1}{A'-A}\left( P - \frac{A+A'}{2}\right)$$ sends the line segment $AA'$ to the line segment joining $-\frac12$ to $\frac12$. Multiply $B'-B$ will rotate and scale the line segment to a parallel copy of $BB'$. After another translation, the map
$$P \mapsto P' \stackrel{def}{=} \frac{B+B'}{2} + \frac{B'-B}{A'-A}\left( P - \frac{A+A'}{2}\right)$$ sends line segment $AA'$ to line segment $BB'$.
To send line segment $BB'$ to an arc joining them, you first figure out the mid point $M$ of the circular arc $BB'$ and let $N = 2O - M$ be the antipodal point of $M$ on the circle holding the arc. $$N = O - \frac{2r}{\sqrt{4r^2 - |B-B'|^2}} \left(\frac{B+B'}{2} - O\right)$$
If you perform a circle inversion with respect to $N$ and radius $R = |B - N| = |B'-N|$, the point $P'$ will get mapped to a point $P''$ on the circular arc joining $BB'$.
$$P' \mapsto P'' \stackrel{def}{=} N + \frac{R^2}{\bar{P}' - \bar{N}}$$ Please note that in denominator of above expression, you need to take complex conjugation for $P'$ and $N$.