Substituting variables $(\vec{r}_i,\vec{r}_j,\vec{r}_k) = (A,B,C)$, with all z-coordinates equal to zero, into the 'in-class' formula gives the following:
$$\vec{r}_c = A+\frac{({\lVert C-A \rVert}^2(B-A)-{\lVert B-A \rVert}^2(C-A)) \times (C-A) \times (B-A)}{2{\lVert (C-A) \times (B-A) \rVert}^2}$$
$$\vec{r}_c = A+\frac{(({\lVert A \rVert}^2+{\lVert C \rVert}^2-2A \cdot C)(B-A)-({\lVert A \rVert}^2+{\lVert B \rVert}^2-2A \cdot B)(C-A)) \times (C-A) \times (B-A)}{2{\lVert (C-A) \times (B-A) \rVert}^2} $$
$$\vec{r}_c = A+\frac{\left({\lVert A \rVert}^2 (B-C) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A) -2\left[(A\cdot C)(B-A)-(A\cdot B)(C-A)\right]\right) \times (C-A) \times (B-A)}{2{\lVert (C-A) \times (B-A) \rVert}^2} $$
Taking a hint from the denominator, I assume the cross product follows the following order. (The terms to the right combine to form a unit vector.)
$$\vec{r}_c = A+\frac{{\lVert A \rVert}^2 (B-C) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A) -2\left[(A\cdot C)(B-A)-(A\cdot B)(C-A)\right]}{2{\lVert (C-A)\times(B-A) \rVert}} \times \frac{(C-A)\times(B-A)}{\lVert (C-A)\times(B-A)\rVert}$$
$$\vec{r}_c = A+\left(\frac{{\lVert A \rVert}^2 (B-C) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A)}{2(A_x(B_y-C_y)+B_x(C_y-A_y)+C_x(A_y-B_y))}-\frac{(A\cdot C)(B-A)-(A\cdot B)(C-A)}{\lVert (C-A)\times(B-A)\rVert}\right)\times \hat z$$
Let $Y=B-A$, $Z=C-A$, $A\cdot B=X\cdot Y$, and $A\cdot C=X\cdot Z$, giving the following:
$$\vec{r}_c = A+\left(\frac{{\lVert A \rVert}^2 (B-C) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A)}{2(A_x(B_y-C_y)+B_x(C_y-A_y)+C_x(A_y-B_y))}-\frac{(X\cdot Z)Y-(X\cdot Y)Z}{\lVert (C-A)\times(B-A)\rVert}\right)\times \hat z$$
$$\vec{r}_c = A+\left(\frac{{\lVert A \rVert}^2 (B-C) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A)}{2(A_x(B_y-C_y)+B_x(C_y-A_y)+C_x(A_y-B_y))}-\frac{X\times(Y\times Z)}{\lVert (C-A)\times(B-A)\rVert}\right)\times \hat z$$
$$\vec{r}_c = A+\left(\frac{{\lVert A \rVert}^2 (B-C) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A)}{2(A_x(B_y-C_y)+B_x(C_y-A_y)+C_x(A_y-B_y))}+X\times\frac{(C-A)\times(B-A)}{\lVert (C-A)\times(B-A)\rVert}\right)\times \hat z$$
$$\vec{r}_c = A+(X\times\hat z)\times\hat z +\frac{{\lVert A \rVert}^2 (C-B) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A)}{2(A_x(B_y-C_y)+B_x(C_y-A_y)+C_x(A_y-B_y))}\times\hat z$$
Now, if I knew how to solve for $-X$, I'd show the following:
$$\vec{r}_c = \frac{{\lVert A \rVert}^2 (C-B) + {\lVert B \rVert}^2 (A-C) + {\lVert C \rVert}^2 (B-A)}{2(A_x(B_y-C_y)+B_x(C_y-A_y)+C_x(A_y-B_y))}\times\hat z$$
In other words, the two formulas are off by a factor of -1, and it appears to be the Wikipedia one that maps improperly, as I had been 'fixing' it by setting yc=-yc
-- the x-axis is symmetric, so I didn't notice that it was also flipped.
The 'in-class' formula needs to be implemented like this: (@Chris Taylor hit it on the head!)
C = [I,0]+(cross((norm(K-I))^2*[J-I,0]-(norm(J-I))^2*[K-I,0],cross([K-I,0],[J-I,0])))/(2*(norm(cross([K-I,0],[J-I,0])))^2);
C = C(1,1:2);
I'll have to try it out later, as it's time to head to class!
It is an application of cross product, since
$$|\vec v \times \vec w|=|\vec v||\vec w|\sin \theta$$
and the area of triangle with sides $|\vec v|$ and $|\vec w|$ is given by
$$A=\frac12|\vec v||\vec w|\sin \theta$$
Note that it not necessary to take into account the order if we consider the absolute value.
For the calculation you should consider for example
$\vec v=(A_x-B_x,A_y-B_y,A_z-B_z)$
$\vec w=(A_x-C_x,A_y-C_y,A_z-C_z)$
$$\vec v \times \vec w=\begin{vmatrix}
i&j&k\\A_x-B_x&A_y-B_y&A_z-B_z\\A_x-C_x&A_y-C_y&A_z-C_z
\end{vmatrix}=...$$
Take also a look here how to calculate area of 3D triangle?
Best Answer
This is not too hard to see using ordinary geometry.
Let $A$, $B$ be fixed points on circle with center $O$, and $C$ any other point on the circle. In triangle $ABC$, bisecting $CB$, $AB$ at $E$, $F$, and joining $AE$, $CF$, then $G$ is the centroid of $\triangle ABC$.
In fixed triangle $AOB$, bisect $AO$ at $D$, and join $BD$, $OF$, giving $K$ the centroid of $\triangle AOB$. Finally, join $GK$.
Assuming as well known, that the centroid divides the median lines of a triangle in a $\frac{2}{1}$ ratio, then$$\frac{CG}{GF}=\frac{OK}{KF}=\frac{2}{1}$$Hence$$\frac{CF}{GF}=\frac{OF}{KF}=\frac{3}{1}$$Therefore$$GK\parallel CO$$whence$$\triangle CFO\sim\triangle GFK$$and$$\frac{CO}{GK}=\frac{3}{1}$$And since $CO$ has a fixed length for all positions of $C$, so does $GK$.
Therefore, since $K$ is fixed in position, $G$ lies always on the circumference of a circle centered on $K$.