There is a famous book of Polya ("How to solve it"), in which the problem of inscribing a square in a triangle is treated in a really interesting way, I strongly suggest the reading.
The inscribed square is clearly unique once we choose the triangle side where two vertices of the square lie. If we suppose that the square has two vertices on $AB$ and side $l$, then:
$$l+l\cot A + l\cot B = c,$$
so:
$$ l = \frac{c}{1+\cot A+\cot B} = \frac{2R \sin A \sin B \sin C}{\sin C + \sin A\sin B}=\frac{abc}{2Rc+ab},$$
where $R$ is the circumradius of $ABC$. In order to maximize $l$, you only need to minimize $2Rc+ab = 2R\left(c+\frac{2\Delta}{c}\right)$, or "land" the square on the side whose length is as close as possible to $\sqrt{2\Delta}$, where $\Delta$ is the area of $ABC$.
For any point $P$ on unit sphere, let $P'$ be its antipodal point and $\hat{P}$ the corresponding unit vector. We will use the notation
$\mathcal{C}_{P_1P_2\cdots P_n}$ to denote a spherical arc starting from $P_1$, passing through $P_2,\ldots$ and end at $P_n$.
Let $A, B, C$ be any three points on unit sphere, close enough to fit
within half of a hemisphere
(i.e. a spherical lune of angle $\frac{\pi}{2}$ ).
Let $\Omega_{ABC}$ be the area of spherical $\triangle ABC$.
It can be computed using a formula by Oosterom and Strackee
$$\tan\left(\frac{\Omega_{ABC}}{2}\right) = \frac{ \left|\hat{A}\cdot (\hat{B} \times \hat{C})\right|}{1 + \hat{A}\cdot\hat{B} + \hat{B}\cdot\hat{C} + \hat{C}\cdot\hat{A}}$$
In the special case where $A, B$ lies on the equator, symmetric with respect to $x$-axis and $C$ lies on the upper hemisphere, i.e.
$$
\begin{cases}
A &= (\cos\alpha,-\sin\alpha,0),\\
B &= (\cos\alpha,+\sin\alpha,0),\\
C &= (x, y, z)\end{cases}
\quad\text{ where } \alpha \in (0,\frac{\pi}{2}), z > 0$$
Above formula reduces to
$$\tan\left(\frac{\Omega_{ABC}}{2}\right) = \frac{\sin\alpha z}{\cos\alpha + x}$$
This implies the locus of $P$ in upper hemisphere for fixed
$\Omega_{ABP} = \Omega_{ABC}$ is the circular arc $\mathcal{C}_{B'CA'}$.
Let $\theta$ be the angle between $\mathcal{C}_{B'CA'}$ and $\mathcal{C}_{B'ABA'}$ at $B'$. The plane holding the locus has the form
$$t ( c + x ) - s z = 0\quad\text{ where }\quad
\begin{cases}
t &= \tan\left(\frac{\Omega_{ABC}}{2}\right)\\
c &= \cos\alpha\\
s &= \sin\alpha
\end{cases}$$
The normal vector of the plane is pointing along the direction $(t, 0, -s)$. This means the tangent vector of $\mathcal{C}_{B'CA'}$ at $B'$ is along the direction $(t, 0, -s ) \times ( -c, -s, 0 ) \propto (s, -c, t)$.
Notice the tangent vector of $\mathcal{C}_{B'ABA'}$ at $B'$
is pointing along the direction $(s,-c,0)$, we find
$$\cos\theta = \frac{s^2 + c^2 + 0}{\sqrt{s^2+c^2}\sqrt{s^2+c^2+t^2}} = \cos\left(\frac{\Omega_{ABC}}{2}\right)$$
From this, we can deduce the circular arcs
$\mathcal{C}_{B'PA'}$ and $\mathcal{C}_{B'ABA'}$ intersect at an angle $\frac{\Omega_{ABC}}{2}$.
This leads to following construction of the desired "spherical centroid" $X$.
Construct the circular arcs $\mathcal{C}_{B'CA'}$ and $\mathcal{C}_{B'ABA'}$,
Trisect the angle $\angle AB'C$ - i.e. find a circular arc
$\mathcal{B'\tilde{C}A'}$ such that $\tilde{C}$ is lying on same side as $C$ with respect to $AB$ and $\angle AB'\tilde{C} = \frac13 \angle AB'C$.
Repeat above procedures for other two sides of $\triangle ABC$ to get
circular arcs $\mathcal{C}_{C'\tilde{A}B'}$ and $\mathcal{C}_{A'\tilde{B}C'}$.
$X$ will be lying on the common intersection of the three circular arcs
$\mathcal{C}_{B'\tilde{C}A'}$, $\mathcal{C}_{C'\tilde{A}B'}$ and $\mathcal{C}_{A'\tilde{B}C'}$.
Best Answer
EDIT.
I'm inserting here a purely geometrical solution, the original reasoning can be seen at the end.
I'll repeatedly make use of the following result: if we have a line $r$ and an arc of circle $\gamma$, and the point $Q\in\gamma$ which is the farthest from $r$ is not an endpoint of $\gamma$, then the tangent at $Q$ is parallel to $r$.
The vesica piscis is composed of two arcs, with centers $O$ and $O'$. Let's consider the case when the vertices $ABC$ of an inscribed triangle are all internal points of the arcs: $A$ on arc $O$ and $BC$ on arc $O'$. If $ABC$ is the triangle of maximum area, then $B$ must be the point on the arc which is the farthest from line $AC$ and by the above result the tangent at $B$ must be parallel to $AC$, i.e. radius $O'B$ must be perpendicular to $AC$ and thus lies on an altitude of triangle $ABC$. By the same argument we also get $O'C\perp AB$, hence $O'C$ also lies on an altitude and $O'$ is the orthocenter of the triangle.
But we also have $OA\perp BC$, implying that altitude $OA$ must pass through $O'$, and that can happen only if $A=O'$. Triangle $ABC$ is then isosceles and right angled at $A$, as shown in figure below. Its area is: $$ area_1={1\over2}d^2. $$
Let's now consider the case when one of the vertices (e.g. $A$) lies at an endpoint of the arcs, while $B$ and $C$ are internal, each on a different arc as in figure below (it's easy to show that we don't get maximum area if $B$ and $C$ lie on the same arc). Moreover, let $A'$ be the other endpoint and $M$ the midpoint of $OO'$ (see figure below).
For the area to be maximum we need as before $O'B\perp AC$ and $OC\perp AB$. If we set $\alpha=\angle O'OC$ we can notice that triangles $OFD$ and $AFM$ have both a right angle and a pair of equal vertical angles, hence $\angle MAF=\angle DOF=\alpha$. Moreover, $\angle O'AC={1\over2}\angle O'OC={1\over2}\alpha$, because they are an inscribed and central angle subtending the same arc in circle $O'$.
We can analogously set $\alpha'=\angle OO'B$ to find: $\angle GAM=\alpha'$ and $\angle OAB={1\over2}\alpha'$. But $\angle OAM=\angle O'AM=30°$, hence we have: $$ \alpha+{1\over2}\alpha'=30° \quad\text{and}\quad \alpha'+{1\over2}\alpha=30°. $$ This system of equation can be easily solved to get: $$ \alpha=\alpha'=20°. $$
Hence the largest triangle in this case is isosceles and symmetric about line $AA'$. To compute its sides, note that from $\angle ABA'=120°$ we get $\angle BA'A=60°-\alpha$ and $AB=AC=2d\sin(60°-\alpha)$. Its area is then $$ area_2={1\over2}AB\cdot AC\cdot\sin(\alpha+\alpha')= {1\over2}\big(2d\sin(60°-\alpha)\big)^2\sin2\alpha= 2d^2\sin^340°\approx 0.53 d^2. $$
Finally, we can consider the case when two points (e.g. $A$ and $B$) lie on the endpoints of the arcs. In that case point $C$ must be either $O$ or $O'$, for the tangent at $C$ to be parallel to $AB$. The area of triangle $ABC$ is: $$ area_1={1\over2}d^2\sin120°={\sqrt3\over4}d^2. $$
In summary, the largest area occurs in the second case.
ORIGINAL REASONING.
Assume WLOG that points $A$ and $B$ are chosen on the right side of the vesica, while $C$ is on the left side (obviously we cannot have maximum area if all three points lie on the same side). For given positions of $A$ and $B$ we get the maximum area if the tangent at $C$ is parallel to $AB$, i.e. if $OC\perp AB$, where $O$ is the center of the right circle (see figure below).
Hence if we choose $C$ at will and some point $D$ on $OC$, points $A$, $B$ of the largest triangle will be the intersections between the left circle (of center $O'$) and the perpendicular to $OC$ at $D$. Note that point $D$ must be close enough to $O$ so that point $A$ won't lie past the intersection point of the circle.
Let $H$ be the projection of $O'$ on $AB$ and $d$ the radius of the circles. If we set $\alpha=\angle O'OC$ and $x=OD$, then by triangle similitude we get: $$ O'H:x=\left({d-{x\over\cos\alpha}}\right):{x\over\cos\alpha} $$ that is: $$ O'H=d\cos\alpha-x. $$ We can then find base $AB$ of the triangle: $$ AB=2\sqrt{d^2-O'H^2}=2\sqrt{d^2-(d\cos\alpha-x)^2}. $$ The area of triangle $ABC$ is then: $$ area={1\over2}AB\cdot CD=(d-x)\sqrt{d^2-(d\cos\alpha-x)^2}. $$ If $x$ is kept constant we get the largest area when $\cos\alpha$ has the lowest possible value, i.e. when $\alpha$ has the greatest possible value (note that $0\le\alpha\le60°$ and $\cos\alpha$ is then positive and decreasing). But the largest possible value is attained by $\alpha$ when point $A$ lies at the intersection of the circles: in the following we can then study only that case.
Let's suppose then that $A$ lies at an intersection of the circles and name $A'$ the other intersection point (see figure below). Note that $\angle A'AB=\alpha$ and $\angle ABA'=120°$, so that $\angle AA'B=60°-\alpha$. We can then compute: $$ AB=2d\sin(60°-\alpha). $$ From the equality $AB=2\sqrt{d^2-(d\cos\alpha-x)^2}$ we then get: $$ x=d\cos\alpha-d\cos(60°-\alpha) $$ and we we can finally write an expression for the area as a function of $\alpha$: $$ area={1\over2}AB\cdot (d-x)=d^2\sin(60°-\alpha) (1-\cos\alpha+\cos(60°-\alpha)). $$ Expanding, this can be rewritten as: $$ {area\over d^2}={\sqrt3\over2}\cos\alpha-{1\over2}\sin\alpha +\cos\alpha\sin\alpha-{\sqrt3\over4}. $$
Now we only need to show that the maximum of this expression is attained for $\alpha=20°$. Note that it vanishes for $\alpha=60°$ and $\alpha=-120°$, hence there must be an absolute maximum between those values, corresponding to a stationary point.
Differentiating the above expression with respect to $\alpha$ and equating the result to zero, we get the equation: $$ 2\cos^2\alpha-{1\over2}\cos\alpha-1={\sqrt3\over2}\sin\alpha. $$ Squaring both sides we obtain $$ (2\cos^2\alpha-{1\over2}\cos\alpha-1)^2={3\over4}(1-\cos^2\alpha), $$ that is: $$ 16c^4-8c^3-12c^2+4c+1=0, $$ where I set $c=\cos\alpha$. This equation has $c={1\over2}$ as solution, which doesn't correspond to an acceptable value for $\alpha$. Hence we can divide the l.h.s. by $2c-1$ to get an equivalent cubic equation: $$ 8c^3-6c-1=0. $$ But from the cosine triplication formula: $\cos3x=4\cos^3x-3\cos x$ we get: $$ 4\cos^320°-3\cos20°=\cos60°={1\over2} $$ implying that $c=\cos20°$ is a solution of the above equation. The other two solutions can be shown to be negative, hence the only acceptable stationary point is $\alpha=20°$, as it was to be proved.