I think the best way to do this would be to look at the sphere as a subset of 3-dimensional space, find the coordinates of A, B, and X in that 3-dimensional space (that's just converting spherical coordinates to Cartesian), and then you can (orthogonally) project everything onto the plane containing A, B, and X. A, B, and X project to themselves, of course, and the arc between A and B will project to the line between A and B, and the arc between X and Y will project to the line between X and Y. Therefore, Y will project to the orthogonal projection of X onto the line AB, and then from that you can just project back onto the sphere and convert that to latitude and longitude. It's somewhat indirect, but it avoids having to work with distances in spherical coordinates.
There are many ways to find your desired point. Here is one way. Since you calculated a length as an approximation rather than exactly, I'll also use approximations for irrational values.
First, let's find the distance $d$ of the desired point from our "base point" $(2,3)$. The base point $(2,3)$, the midpoint of the line segment between the two given points, and the desired point form a $30°$-$60°$-$90°$ triangle. The short leg has half the length of the line segment that you already calculated. Let's call that $\frac L2$, where $L$ is your calculated length. We then use the $30°$ angle in the right triangle to find the desired distance $d$:
$$\cos 30°=\frac{\frac L2}{d}$$
$$d=\frac L{2\cos 30°}=1.82574185835$$
Now we want to find the angle of inclination from the base point to the desired point. The angle of inclination from $(2,3)$ to $(5,2)$ is $\tan^{-1}\frac{-1}3=-18.4349488229°$. We add $30°$ to that to get our desired angle of inclination $\theta=11.5650511771°$.
We now find the desired point by its position relative to the base point.
$$\begin{align}
(x,y) &= (2+d\cos\theta,3+d\sin\theta) \\
&= (3.78867513459,3.36602540378)
\end{align}$$
This answer checks in Geogebra.
As I state in a comment to this answer, the method above for finding $d$, the distance of the desired point from the base point, works only for isosceles triangles where you know the endpoints of the base as well as the angles there. If the angles at the two known vertices differ, there is another way.
Let's say the length of the known triangle side is $L$, the known angle at the base point is $\alpha$ and the angle at the other known vertex is $\beta$. We can use the law of sines to get
$$\frac{\sin(180°-\alpha-\beta)}{L}=\frac{\sin(\beta)}{d}$$
so
$$d=\frac{L\sin(\beta)}{\sin(180°-\alpha-\beta)}$$
The rest of my method above works after this. This more general method also works and checks in your particular problem. Let me know if you want me to edit my answer above to only show the more general method: that might be easier for you.
Best Answer
The notation of the diagram could be improved. We really cannot call the points $A(x,y)$ and $B(x,y)$, since their coordinates are different.
Before you try for a "general" procedure, let's work out the details for a specific choice of $A$ and $B$, say $A(1,2)$ and $B(7,10)$.
Then the midpoint $M$ of the line segment $AB$ has coordinates $(\frac{1+7}{2}, \frac{7+10}{2})=(4,6)$. I cheated a bit to make the numbers nice.
The line $AB$ has slope $8/6$. So the line $MC$ has slope the negative reciprocal of $8/6$, which is $-6/8$, by perpendicularity.
Now you can find the equation of the line $MC$. The usual method gives that this equation is $$\frac{y-6}{x-4}=-\frac{6}{8}.$$
The equation can be rewritten in various ways, like $y=(-3/4)x+9$. Now for a specific $d$, (pick one, like $d=8$) you want the distance from $M(4,6)$ to the point $(x, (-3/4)x+9)$ on the line to be $d$. Use the "square of distance" formula, also known as the Pythagorean Theorem. You will get a quadratic equation in $x$.
Solve. There will be in general two solutions, as your picture makes clear.
Now if you really want to develop a general formula, give general names to the coordinates of $A$ and of $B$, and use exactly the same procedure. Things will look fairly messy, because of all the symbols running around, but in principle you are doing nothing really different from what we did in the specific numerical case.
Later, when you learn about vectors, there will be a cleaner-looking way of handling the problem.
Added: I had initially dismissed as not very practical the notion of using angles. Though the method described above is easier, the idea of your first diagram can be used to produce an answer. Since we know $d$ and can easily find half the distance between $A$ and $B$, we can compute $\tan\alpha$. We can also compute the tangent of the angle between $AB$ and the vertical line through $A$, and so we can find the slope, and then the equation, of $AC$. We also know the equation of what I called the line MC$. Now we have two linear equations, and can find where the two lines meet.