Don't write down too many equations to be solved, but produce the desired center ${\bf c}=(a,b)$ in a forward movement instead. Let ${\bf z}_i=(x_i,y_i)$ $\ (i=0,1)$ be the two given points, put $\epsilon:=1$ if the arc should go from ${\bf z}_0$ to ${\bf z}_1$ counterclockwise, and put $\epsilon:=-1$ otherwise.
Next, let $d:=|{\bf z_1}-{\bf z_0}|=\sqrt{(x_1-x_0)^2+(y_1-y_0)^2}$ be the distance and ${\bf m}:=\bigl({x_0+x_1 \over2}, {y_0+y_1\over2}\bigr)$ be the midpoint of ${\bf z_0}$ and ${\bf z_1}$. Then
$${\bf n}:=(u,v):=\Bigl({x_1-x_0\over d},{y_1-y_0\over d}\Bigr)$$
is the unit normal in the direction ${\bf z_1}-{\bf z}_0$, and ${\bf n}^*:=(-v,u)$ is the unit vector you get by rotating ${\bf n}$ counterclockwise by $90^\circ$.
Given $r>0$ the center ${\bf c}$ has a distance $h:=\sqrt{r^2 -d^2/4}$ from ${\bf m}$, and the given $\epsilon$ together with ${\bf n}^*$ tell us in which direction we should go. In vectorial notation the center is given by
$${\bf c}\ =\ {\bf m}+\epsilon\ h\ {\bf n}^*\ ,$$
so that coordinate-wise we get
$$a={x_0+x_1 \over2}-\epsilon\ h\ v, \qquad b={y_0+y_1 \over2}+\epsilon\ h\ u\ .$$
Another way, using vectors:
Let $\mathbf v = (x_1,y_1)-(x_0,y_0)$. Normalize this to $\mathbf u = \frac{\mathbf v}{||\mathbf v||}$.
The point along your line at a distance $d$ from $(x_0,y_0)$ is then $(x_0,y_0)+d\mathbf u$, if you want it in the direction of $(x_1,y_1)$, or $(x_0,y_0)-d\mathbf u$, if you want it in the opposite direction. One advantage of doing the calculation this way is that you won't run into a problem with division by zero in the case that $x_0 = x_1$.
Best Answer
$(A, B)$ is a normal vector for the line, therefore $v = (-B, A)$ is a direction vector and you get all points on the line with
$$ (x, y) = P_0 + t \, v = (X_0, Y_0) + t (-B, A), \quad t \in \mathbb R. $$
Now choose $t$ such that the length of $t(−B,A)$ is equal to the given distance $d$, this gives the two points
$$ (X_2, Y_2) = (X_0, Y_0) \pm \frac d{\sqrt { A^2 + B^2}} (-B, A) \quad . $$