WLOG, say the center of the circle ($O$) is at the origin. Vertices of the pentagon $ABCDE$ are represented by position vectors $\overline{a}, \overline{b}, \overline{c}, \overline{d}$ and $\overline{e}$.
Centroid of $\triangle ABC, \, \overline {g} = \frac{\overline{a} + \overline{b} + \overline{c}}{3}$
Line $DE = \overline{d} - \overline{e}$
As points $A, B, C, D, E$ are concyclic with center at $O$
$|\overline{a}|^2 = |\overline{b}|^2 = |\overline{c}|^2 = |\overline{d}|^2 = |\overline{e}|^2$ ...(i)
If a point $P$ with position vector $\overline{p} \,$ is on the perpendicular line from the centroid of $\triangle ABC$ to the line $DE$,
$(\overline{p}-\overline{g}) \cdot (\overline{d} - \overline{e}) = 0$
Based on (i) one of the ways for the dot product to be zero is
$(\overline{p}-\overline{g}) = n_1 (\overline{d}+\overline{e}) \,$ (you can easily show why $\overline{p} = \overline{g}$ will not give you the concurrent point by symmetry)
$\overline{p}-\overline{g} = \overline{p}-\frac{\overline{a} + \overline{b} + \overline{c}}{3} = n_1 (\overline{d}+\overline{e})$ ...(ii)
Similarly,
$\overline{p}-\frac{\overline{b} + \overline{c} + \overline{d}}{3} = n_2 (\overline{e}+\overline{a})$ ...(iii)
From (ii)-(iii), you get one solution when $n_1 = n_2 = \frac{1}{3}$ and
$\overline {p} = \frac{\overline{a} + \overline{b} + \overline{c} + \overline{d} + \overline{e}}{3}$
Now we need to prove this point is the point of concurrency for other $3$ lines too. So we take the lines from centroids of $\triangle CDE, \triangle DEA, \triangle EAB$ through point $\overline {p}$ and show each of them is perpendicular to the line segment made by other two vertices.
$(\overline{p}- \frac{\overline{c} + \overline{d} + \overline{e}}{3}) \cdot (\overline{a} - \overline{b}) = 0$
$(\overline{p}- \frac{\overline{d} + \overline{e} + \overline{a}}{3}) \cdot (\overline{b} - \overline{c}) = 0$
$(\overline{p}- \frac{\overline{e} + \overline{a} + \overline{b}}{3}) \cdot (\overline{c} - \overline{d}) = 0$
which is easy to show given (i).
The unit normal vector is $\vec{n}=\frac{1}{\sqrt{6}}\langle 1,-1,2\rangle$ and the vector $\overrightarrow{AB}=\langle 1,1,0\rangle$.
Using Rodrigue's rotation formula, the rotation of $\overrightarrow{AB}$ about $\vec{n}$ by $\pm 60^\circ$ is
$$\overrightarrow{AC}=\frac{1}{2}\overrightarrow{AB}\pm\frac{\sqrt{3}}{2}\vec{n}\times\overrightarrow{AB}+\frac{1}{2}\vec{n}(\vec{n}\cdot \overrightarrow{AB})$$
Since we have
$$\vec{n}\times\overrightarrow{AB}=\frac{1}{\sqrt{6}}\langle -2,2,2\rangle$$
$$\vec{n}\cdot\overrightarrow{AB}=0$$
Our equation for $\overrightarrow{AC}$ simplifies to
\begin{align}
\overrightarrow{AC}
&=\frac{1}{2}\langle 1,1,0\rangle\pm\frac{\sqrt{2}}{2}\langle -1,1,1\rangle\\
&=\frac{1}{2}\langle 1,1,0\rangle -\frac{\sqrt{2}}{2}\langle -1,1,1\rangle\\
&=\boxed{\frac{1}{2}\left\langle 1+\sqrt{2},1-\sqrt{2},-\sqrt{2}\right\rangle}
\end{align}
Best Answer
This is a fairly well known algorithm. It all comes down to using the cross product. Define the vectors $AB$, $BC$ and $CA$ and the vectors $AP$, $BP$ and $CP$. Then $P$ is inside the triangle formed by $A, B$ and $C$ if and only if all of the cross products $AB\times AP$, $BC\times BP$ and $CA\times CP$ point in the same direction relative to the plane. That is, either all of them point out of the plane, or all of them point into the plane.
Update:
this test can be performed using dot products.Update 2: As emphasised in the comments, you only have to compare signs of the third components.