Don't write down too many equations to be solved, but produce the desired center ${\bf c}=(a,b)$ in a forward movement instead. Let ${\bf z}_i=(x_i,y_i)$ $\ (i=0,1)$ be the two given points, put $\epsilon:=1$ if the arc should go from ${\bf z}_0$ to ${\bf z}_1$ counterclockwise, and put $\epsilon:=-1$ otherwise.
Next, let $d:=|{\bf z_1}-{\bf z_0}|=\sqrt{(x_1-x_0)^2+(y_1-y_0)^2}$ be the distance and ${\bf m}:=\bigl({x_0+x_1 \over2}, {y_0+y_1\over2}\bigr)$ be the midpoint of ${\bf z_0}$ and ${\bf z_1}$. Then
$${\bf n}:=(u,v):=\Bigl({x_1-x_0\over d},{y_1-y_0\over d}\Bigr)$$
is the unit normal in the direction ${\bf z_1}-{\bf z}_0$, and ${\bf n}^*:=(-v,u)$ is the unit vector you get by rotating ${\bf n}$ counterclockwise by $90^\circ$.
Given $r>0$ the center ${\bf c}$ has a distance $h:=\sqrt{r^2 -d^2/4}$ from ${\bf m}$, and the given $\epsilon$ together with ${\bf n}^*$ tell us in which direction we should go. In vectorial notation the center is given by
$${\bf c}\ =\ {\bf m}+\epsilon\ h\ {\bf n}^*\ ,$$
so that coordinate-wise we get
$$a={x_0+x_1 \over2}-\epsilon\ h\ v, \qquad b={y_0+y_1 \over2}+\epsilon\ h\ u\ .$$
Welcome to Math.SE! Here is my sketch for the case of clockwise direction: initial point is $A$, the endpoint you want is $B$, and $C$ is the radius of the circle on which the arc lies.
I will use polar angles: you can see the description in Wikipedia. The formulas for conversion from polar to Cartesian (on the same wiki article) will also be used.
On the picture, $\theta$ is the polar angle of the direction in which the curve departs from point $A$. Since any radius of a circle is perpendicular to the circle, the polar angle of the vector $\vec{CA}$ is $\theta+\pi/2$. The length of $\vec{CA}$ is $r$. Therefore, its Cartesian coordinates are
$$\vec {CA} = ( r\cos(\theta+\pi/2), r\sin(\theta+\pi/2)) = ( -r\sin(\theta), r\cos(\theta)) $$
Next, we need the coordinates of the vector $\vec{CB}$. Its length is also $r$. Since $\angle ACB$ is $L/r$ radian, the polar angle of $\vec{CB}$ is $\theta+\pi/2-L/r$. Convert to Cartesian:
$$\vec {CB} = ( r\cos(\theta+\pi/2-L/r), r\sin(\theta+\pi/2-L/r)) = ( -r\sin(\theta-L/r), r\cos(\theta-L/r)) $$
Finally, $\vec{AB}=\vec{CB}-\vec{CA}$, which yields
$$\boxed{\vec {AB} = ( -r\sin(\theta-L/r)+r\sin(\theta), r\cos(\theta-L/r)-r\cos(\theta)) } $$
These can be rewritten using some trigonometric identities, but I don't think it would win anything. As a sanity check, consider what happens when $L=0$: the vector is zero, hence $B$ is the same as $A$. As an aside, if $r\to \infty$ the curve becomes a straight line segment, but figuring out the limit is an exercise in calculus. :-)
If the curve bends counterclockwise, the signs will be different in a few places. Namely, the polar angle of $\vec{CA}$ will be $\theta-\pi/2$, hence
$$\vec {CA} = ( r\sin(\theta), -r\cos(\theta)) $$
The polar angle of $\vec{CB}$ will be $\theta-\pi/2+L/r$, hence
$$\vec {CB} = ( r\sin(\theta+L/r), -r\cos(\theta+L/r)) $$
The conclusion in this case is
$$\boxed{\vec {AB} = ( r\sin(\theta+L/r)-r\sin(\theta), -r\cos(\theta+L/r)+r\cos(\theta))}$$
Later: a simpler solution for the case when $C$ is given. First, calculate the vector $\vec{CA}$ and convert it to polar coordinates using these formulas. Then either increase or decrease the angle by $L/r$, depending on counterclockwise/clockwise choice.
Since you wanted JavaScript, I made a jsfiddle and also copied the code below. The parameters are coordinates of A and C, as well as length of the arc and the direction. The radius $r$ is calculated within the function.
function findB(Ax, Ay, Cx, Cy, L, clockwise) {
var r = Math.sqrt(Math.pow(Ax - Cx, 2) + Math.pow(Ay - Cy, 2));
var angle = Math.atan2(Ay - Cy, Ax - Cx);
if (clockwise) {
angle = angle - L / r;
}
else {
angle = angle + L / r;
}
var Bx = Cx + r * Math.cos(angle);
var By = Cy + r * Math.sin(angle);
return [Bx, By];
}
document.write(findB(0, 1, 1, 0, 1, true));
Best Answer
For 2D case, you can determine the arc's mid-point $Q$ with the help of the following picture and formula:
$M=(P_1+P_2)/2$ ,
Bulge = $D_1/D_2=tan(Delta/4)$
$\vec{MQ}$ is perpendicular to $\vec{P_1P_2}$
For 3D case, you cannot uniquely determine the arc from two 3D points and bulge as there will be infinite number of solutions that share the same end points and the same bulge. For example, the green arc in above picture can rotate around the axis $P_1P_2$ by any angle and that still satisfy your input.