Given points $A(1, 0, 0), B(0, 1, 0), C(0, 0, 1)$ in barycentric coordinates, and points $P(x_P, y_P, z_P), Q(x_Q, y_Q, z_Q), R(x_R, y_R, z_R)$, what would the barycentric coordinates of the circumcenter of $\triangle PQR$ be? The circumcircle can be found in a straightforward manner by using the equation for a circle in barycentric coordinates.
[Math] Barycentric Coordinates of the circumcenter of an arbitrary triangle
barycentric-coordinatescoordinate systemseuclidean-geometrytriangles
Related Solutions
Hint:
using elemental geometrical properties of equilateral triangles you can see that:
$$ PC=PA=PB=\frac{10\sqrt{3}}{3} $$
and the distance of $P$ from the sides is half this value.
Can you do from this adding or subtracting such values from the coordinates of $P$ to find the vertices?
First, a bit of background so that others also understand what this is about.
We have a non-degenerate triangle (one that has nonzero area, so is not a point or a line) with vertices $\vec{p}_1$, $\vec{p}_2$, and $\vec{p}_3$. Any point $\vec{p}$ within this triangle can be specified via barycentric coordinates $u$, $v$, $w$: $$\begin{aligned} \vec{p} &= u \vec{p}_1 + v \vec{p}_2 + w \vec{p}_3 \\ u + v + w &= 1 \\ 0 &\le u \le 1 \\ 0 &\le v \le 1 \\ 0 &\le w \le 1 \\ \end{aligned}$$ This also implies that $w = 1 - u - v$, i.e. $$\vec{p} = u \vec{p}_1 + v \vec{p}_2 + (1 - u - v) \vec{p}_3$$ Reordering the terms above we can also write it equivalently as $$\vec{p} = \vec{p}_3 + u (\vec{p}_1 - \vec{p}_3) + v (\vec{p}_2 - \vec{p}_3)$$ which is equivalent to a linear (but not necessarily orthogonal) coordinate system in $u$,$v$ with origin at $\vec{p}_3$, $u$ axis $\vec{p}_1-\vec{p}_3$, and $v$ axis $\vec{p}_2-\vec{p}_3$. This, in turn, means that the transformation from barycentric coordinate system $\overline{b} = (u, v)$ (with $0 \le u, v, w \le 1$ and $w = 1 - u - v$) to Euclidean 2D coordinate system can be described as $$\begin{aligned} \vec{p} &= \vec{p}_3 + \mathbf{M} \overline{b} \\ \mathbf{M} &= \left[ \begin{matrix} \vec{p}_1 - \vec{p}_3 & \vec{p}_2 - \vec{p}_3 \end{matrix} \right ] \\ \end{aligned}$$ (Note, $\mathbf{M}$ is a matrix with 2 columns, and as many rows as there are dimensions in the Euclidean coordinate system.)
With this, we have accidentally proven that the transform between Euclidean coordinates and barycentric coordinates is an affine transformation, because it is the composition of two functions: a translation and a linear map.
By definition, affine transformations preserve lines and parallelism.
In two dimensions, the inverse mapping is $$\overline{b} = \mathbf{M}^{-1} \left(\vec{p} - \vec{p}_3\right)$$ where $$\mathbf{M} = \left[\begin{matrix} m_{11} & m_{12} \\ m_{21} & m_{22} \end{matrix}\right] \quad \iff \quad \mathbf{M}^{-1} = \left[\begin{matrix} \frac{m_{22}}{m_{11} m_{22} - m_{12} m_{21}} & \frac{- m_{12}}{m_{11} m_{22} - m_{12} m_{21}} \\ \frac{- m_{21}}{m_{11} m_{22} - m_{12} m_{21}} & \frac{m_{11}}{m_{11} m_{22} - m_{12} m_{21}} \end{matrix}\right]$$ In three dimensions, the easiest is to check which component in $(\vec{p}_1 - \vec{p}_3)\times(\vec{p}_2 - \vec{p}_3)$ is the largest in magnitude, and drop that Euclidean coordinate component, and do the above transform using only the two coordinate axes left. It is guaranteed to provide exact correct results, if the triangle is nondegenerate.
If the triangle is degenerate, then the common denominator is zero, $m_{11} m_{22} - m_{12} m_{21} = 0$, and no inverse exists.
There are many different ways to describe lines in barycentric coordinates. Remember, that although there are three coordinates, one of them can always be trivially eliminated ($w = 1 - u - v$), and is usually omitted, especially in computer programs and simulations using barycentric coordinates. (It can be confusing to see barycentric coordinates with only two components, but one only needs to remember that the third is always related to the two via $w = 1 - u - v$, because full barycentric coordinates sum to 1.)
(If we allow full real range in $u$ and $v$ and $w$, with $w = 1 - u - v$, the barycentric coordinates can cover the full plane the non-degenerate triangle is in. This means that if we know the triangle has nonzero area – is not a point or a line –, we can safely use $u$, $v$ coordinates anywhere on the plane, not only inside the triangle. It is then useful to remember that the triangle edges are at $u = 0$, $v = 0$, and $w = 0$ i.e. $u + v = 1$.)
However, affine transforms do not necessarily conserve distance. Even though the distance between the two lines were to stay constant in Euclidean coordinates, just rotating them can change their distance in barycentric coordinates. This is because barycentric coordinates are not orthonormal.
Therefore, I would recommend describing each line in the barycentric coordinates using two points. (This means that if you actually have a point and a direction, you can use the point+direction as the second point, even though it may be outside the triangle.) Then, transform the four points back to Euclidean coordinates. Convert back to point ($\vec{a}_0$ for one line, $\vec{b}_0$ for the other) and direction (difference between the two points, $\vec{a}_\Delta$ for one line, and $\vec{b}_\Delta$ for the other).
The cosine of the angle $\varphi$ between the two lines is $$\cos\varphi = \frac{\left\lvert \vec{a}_\Delta \cdot \vec{b}_\Delta \right\rvert}{\left\lVert\vec{a}_\Delta\right\rVert ~ \left\lVert\vec{b}_\Delta\right\rVert}$$ We take the absolute value of the dot product, because we consider line $\vec{a}_0$,$\vec{a}_\Delta$ the same as $\vec{a}_0$,$\vec{a}_\Delta$. If the cosine is close enough to 1, you can consider them parallel.
When the two lines are parallel, we can solve their distance $d$ by using a temporary variable $t$ so that $\vec{a}_0 + t \vec{a}_\Delta$ is at minimum distance to $\vec{b}_0$. This occurs when their distance reaches an extremum, and the solution is $$t = \frac{\vec{a}_\Delta \cdot \left( \vec{b}_0 - \vec{a}_0 \right) }{\left\lVert \vec{a}_\Delta \right\rVert^2}$$ Then, $$d = \left\lVert \vec{a}_0 + t \vec{a}_\Delta - \vec{b}_0 \right\rVert$$
Depending on how you describe your lines, it is usually possible to combine the above steps, so that fewer actual arithmetic operations are needed as plain reading of the above would indicate. That is, I didn't write the above to be "optimum" in any way; I tried to make it as easy to understand as possible.
I recommend you use a Computer Algebra System such as free Maxima (wxMaxima is the graphical user interfaced one) available for all operating systems to explore the details. In particular, if the direction vectors have unit lengths in Euclidean coordinates, you'll find a lot of the math related to lines becomes much, much simpler. (You'll be able to omit many $\lVert\vec{a}_\Delta\rVert$ and $\lVert\vec{b}_\Delta\rVert$ multipliers and divisors, in particular.)
Best Answer
You want the barycentric coordinates for any triangle: that is, you want $A$, $B$, $C$ to have coordinates $(1,0,0)$, $(0,1,0)$ and $(0,0,1)$ respectively, so that $P$, $Q$, $R$ will have "general" barycentric coordinates.
But you can simplify: saying that $M$ has barycentric coordinates $(\alpha,\beta,\gamma)$ w.r.t. $A,B,C$ means that
$$\overrightarrow{OM}=\frac{\alpha\overrightarrow{OA}+\beta\overrightarrow{OB}+\gamma\overrightarrow{OC}}{\alpha+\beta+\gamma}$$
So, you can get the barycentric coordinates of the circumcenter of $PQR$ w.r.t. $P, Q, R$, which is the formula I linked to above. You get coordinates $(\sin2\hat P,\sin2\hat Q,\sin2\hat R)$, where I denote the angles of triangle $PQR$ by $\hat P,\hat Q,\hat R$.
This means you have, for circumcenter $K$ (not the usual notation for circumcenter, but anyway).
$$\overrightarrow{OK}=\frac{\sin(2\hat P)\ \overrightarrow{OP}+\sin(2\hat Q)\ \overrightarrow{OQ}+\sin(2\hat R)\ \overrightarrow{OR}}{\sin(2\hat P)+\sin(2\hat Q)+\sin(2\hat R)}$$
Or equivalently
$$\overrightarrow{OK}=\alpha\overrightarrow{OP}+\beta\overrightarrow{OQ}+\gamma\overrightarrow{OR}$$
With $\alpha+\beta+\gamma=1$ and
$$\alpha=\frac{\sin(2\hat P)}{\sin(2\hat P)+\sin(2\hat Q)+\sin(2\hat R)}$$ $$\beta=\frac{\sin(2\hat Q)}{\sin(2\hat P)+\sin(2\hat Q)+\sin(2\hat R)}$$ $$\gamma=\frac{\sin(2\hat R)}{\sin(2\hat P)+\sin(2\hat Q)+\sin(2\hat R)}$$
It will be simpler in the following to have sum $1$.
You can further expand this using the barycentric coordinates of $P,Q,R$ w.r.t. $A,B,C$, that is the equations:
$$\overrightarrow{OP}=\frac{x_P\overrightarrow{OA}+y_P\overrightarrow{OB}+z_P\overrightarrow{OC}}{x_P+y_P+z_P}$$
$$\overrightarrow{OQ}=\frac{x_Q\overrightarrow{OA}+y_Q\overrightarrow{OB}+z_Q\overrightarrow{OC}}{x_Q+y_Q+z_Q}$$
$$\overrightarrow{OR}=\frac{x_R\overrightarrow{OA}+y_R\overrightarrow{OB}+z_R\overrightarrow{OC}}{x_R+y_R+z_R}$$
Again, you can admit without loss of generality that $x_P+y_P+z_P=x_Q+y_Q+z_Q=x_R+y_R+z_R=1$, to simplify.
When expanding, you will get a relation of the form
$$\overrightarrow{OK}=\alpha'\overrightarrow{OA}+\beta'\overrightarrow{OB}+\gamma'\overrightarrow{OC}$$
With
$$\alpha'=\alpha x_P+\beta x_Q+\gamma x_R$$ $$\beta'=\alpha y_P+\beta y_Q+\gamma y_R$$ $$\gamma'=\alpha z_P+\beta z_Q+\gamma z_R$$
And it's pretty straightforward to check that $\alpha'+\beta'+\gamma'=1$, so $(\alpha',\beta',\gamma')$ are barycentric coordinates of $K$ w.r.t. $A,B,C$.