The line can be parameterized by $\mathbf{x}(t) = (x_1 + t(x_2 - x_1),y_1 + t(y_2 - y_1))$ for all $t\in\mathbb{R}$. Since point $P_3 = (x_3,y_3)$ is on this line, there exists a $t$ such that $\mathbf{x}(t) = (x_3,y_3)$. Assuming the point $P_3$ comes after the point $P_2$ on the line, we can solve for $t$ by observing that the distance from $P_1$ to $P_3$ is
$$
r + h = \sqrt{(x_3 - x_1)^2 + (y_3 - y_1)^2} = |t|\sqrt{(x_2-x_1)^2 + (y_2 - y_1)^2}
$$
Hence,
$$
t = \frac{r + h}{\sqrt{(x_2-x_1)^2 + (y_2 - y_1)^2}}
$$
Plugging this value of $t$ into $\mathbf{x}(t)$ will give you the coordinates of $P_3$.
I will just give a sketch for now how to solve it, I can fill in more details if necessary.
First define vectors $v_i = (x_i-a,y_i-b),\, i = 1,2,3$. Then we can rewrite the equations as
\begin{align}
\langle v_1, v_2\rangle = a_{12},\ \langle v_2, v_3\rangle = a_{23},\ \langle v_3, v_1\rangle = a_{31},\\
\det[v_1^t\ v_2^t] = b_{12},\ \det[v_2^t\ v_3^t] = b_{23},\ \det[v_3^t\ v_1^t] = b_{31},\\
\end{align} where I changed dot's and det's to a's and b's.
Denote by $\vartheta_{ij}$ the (oriented) angle between vectors $v_i$ and $v_j$. Remember that dot product in $\mathbb R^2$ is given by formula $\|v\|\|w\|\cos\vartheta$ and that determinants above measure (signed) area of parallelogram formed by the vectors, which can also be expressed as $\|v\|\|w\|\sin\vartheta$, and thus the system becomes
\begin{align}
\|v_1\|\|v_2\|\cos\vartheta_{12} = a_{12},\ \|v_2\|\|v_3\|\cos\vartheta_{23} = a_{23},\ \|v_3\|\|v_1\|\cos\vartheta_{31} = a_{31},\\
\|v_1\|\|v_2\|\sin\vartheta_{12} = b_{12},\ \|v_2\|\|v_3\|\sin\vartheta_{23} = b_{23},\ \|v_3\|\|v_1\|\sin\vartheta_{31} = b_{31}.\\
\end{align}
Then, you can find the lengths $\|v_i\|$ by looking at the system $$\|v_1\|\|v_2\| = c_{12},\ \|v_2\|\|v_3\| = c_{23},\ \|v_3\|\|v_1\| = c_{31},$$ where $c_{ij} = \sqrt{a_{ij}^2+b_{ij}^2}$ obtained by squaring the above equations and using $\sin^2t+\cos^2t = 1$. Solving it gives you $$\|v_1\|=\sqrt{\frac{c_{12}c_{31}}{c_{23}}},\ \|v_2\|=\sqrt{\frac{c_{12}c_{23}}{c_{31}}},\ \|v_3\|=\sqrt{\frac{c_{23}c_{31}}{c_{12}}}.$$
The whole thing is, from geometric perspective, obviously rotationally invariant (rotating all of the vectors won't change the angles between them or the areas), so fix some angle $\vartheta$. Then, the solution can be represented as complex numbers as
$$v_1 = \|v_1\|e^{i\vartheta},\ v_2 = \|v_2\|e^{i(\vartheta +\vartheta_{12})},\ v_3 = \|v_3\|e^{i(\vartheta-\vartheta_{31})}.$$
More explicitly, $e^{i\vartheta_{12}} = \cos\vartheta_{12} + i\sin\vartheta_{12} = \frac{a_{12}}{c_{12}}+\frac{b_{12}}{c_{12}}i$ and $e^{i\vartheta_{31}} = \cos\vartheta_{31} + i\sin\vartheta_{31} = \frac{a_{31}}{c_{31}}+\frac{b_{31}}{c_{31}}i$, so we have
\begin{align}v_1 &= \sqrt{\frac{c_{12}c_{31}}{c_{23}}}(\cos\vartheta + i\sin\vartheta),\\
v_2 &= \sqrt{\frac{c_{12}c_{23}}{c_{31}}}(\cos\vartheta + i\sin\vartheta)(\frac{a_{12}}{c_{12}}+i\frac{b_{12}}{c_{12}}),\\
v_3 &= \sqrt{\frac{c_{23}c_{31}}{c_{12}}}(\cos\vartheta + i\sin\vartheta)(\frac{a_{31}}{c_{31}}-i\frac{b_{31}}{c_{31}}),\ \vartheta\in\mathbb R.\end{align}
All you have to do now is expand and $x$'s will be the real parts, while $y$'s the imaginary parts of the above complex number representation.
Note, however, that the system is overdetermined since knowing the angles between $v_1$ and $v_2$ and $v_2$ and $v_3$ will also give you the angle between $v_1$ and $v_3$.
Best Answer
$$\begin{bmatrix} x_1\\ x_2\\ x_3\end{bmatrix} \begin{bmatrix} y_1\\ y_2\\ y_3\end{bmatrix}^T = \begin{bmatrix} p_1 & 1-p_1 & *\\ * & p_2 & 1-p_2\\ 1-p_3 & * & p_3\end{bmatrix}$$
Hence, we have the following matrix completion problem
If the matrix is rank-$1$, then the 2nd and 3rd columns are multiples of the 1st column
$$\begin{array}{rl} 1-p_1 &= \alpha \, p_1\\ p_2 &= \alpha \, t_2\\ t_3 &= \alpha (1 - p_3)\\\\ t_1 &= \beta \, p_1\\ 1-p_2 &= \beta \, t_2\\ p_3 &= \beta (1-p_3)\end{array}$$
Hence,
$$\alpha = \dfrac{1-p_1}{p_1}\qquad \qquad t_2 = \dfrac{p_1 p_2}{1-p_1}\qquad \qquad t_3 = \dfrac{(1-p_1) (1 - p_3)}{p_1}$$
$$t_1 = \dfrac{p_1 p_3}{1-p_3}\qquad \qquad t_2 = \dfrac{(1-p_2) (1 - p_3)}{p_3}\qquad \qquad \beta = \frac{p_3}{1 - p_3}$$
In order to have a solution, we must have
$$\dfrac{p_1 p_2}{1-p_1} = \dfrac{(1-p_2) (1 - p_3)}{p_3}$$
or, in a much nicer form,
$$p_1 p_2 p_3 = (1 - p_1) (1-p_2) (1 - p_3)$$
Completing and factoring the matrix, we find one solution
$$\begin{bmatrix} p_1 & 1-p_1 & \dfrac{p_1 p_3}{1-p_3}\\ \dfrac{p_1 p_2}{1-p_1} & p_2 & 1-p_2\\ 1-p_3 & \dfrac{p_2 p_3}{1 - p_2} & p_3\end{bmatrix} = \begin{bmatrix} p_1\\ \dfrac{p_1 p_2}{1-p_1}\\ \dfrac{p_1 p_3}{1-p_3}\end{bmatrix} \begin{bmatrix} 1\\ \alpha\\ \beta\end{bmatrix}^T$$