(Essentially this question, but in 3 dimensions.)
In 3D space, points $a$ and $b$ are known. In addition, a unit vector $\hat{t}$ is known. Assume $b-a$ and $\hat{t}$ are not parallel.
The goal: determine the center point of a circle which passes through both $a$ and $b$ and is tangent to $\hat{t}$ at point $a$.
(The circle will then lie on the plane spanned by $b-a$ and $\hat{t}$.)
My attempt:
Let $c$ be the center of the circle (the goal).
Get a vector in the direction of the normal of the plane $n$:
$$n=(b-a)\times\hat{t}$$
Find three equations that can be used to solve for $[c_x, c_y, c_z]$.
First, using the fact that $(a-c)$ and $\hat{t}$ must be perpendicular, $(a-c)\cdot\hat{t}=0$, or
$t\cdot c = t \cdot a \tag{1}$
Second, $(b-c)$ must be perpendicular to $n$:
$n \cdot c = n \cdot b \tag{2}$
Third, $c$ must lie on the perpendicular bisecting plane of $a$ and $b$:
$(b-a)\cdot c = (\frac{b-a}{2})\cdot (b-a) \tag{3}$
This gives the equation
$$
\begin{bmatrix}
t_x & t_y & t_z \\
n_x & n_y & n_z \\
b_x-a_x & b_y-a_y & b_z-a_z
\end{bmatrix}
\begin{bmatrix}
c_x \\
c_y \\
c_z
\end{bmatrix}
=
\begin{bmatrix}
t \cdot a \\
n \cdot b \\
(\frac{b-a}{2})\cdot (b-a)
\end{bmatrix}
$$
However, solving that (by plugging in $a$, $b$, and $\hat{t}$ and doing it on a computer) is giving me wildly off results. $c$ is in the correct plane, but that's about it.
Best Answer
With $p = (x,y,z)$ the circle's center is defined by the intersection of the following three planes
$$ \cases{ \Pi_1\to (p-a)\cdot \vec t = 0\\ \Pi_2\to (p-a)\cdot\left((b-a)\times \vec t\right)=0\\ \Pi_3\to (p-\frac 12(a+b))\cdot(b-a) = 0 } $$
giving $c$. Now, the circle can be parametrized as
$$ p = c + r\left(\vec e_1\cos\theta+\vec e_2\sin\theta\right) $$
with
$$ \cases{\vec e_1 = \vec t\\ \vec u = (b-a)-\left((b-a)\cdot \vec t\right)\vec t\\ \vec e_2 = \frac{\vec u}{|\vec u|}\\ r = |a-c| } $$
Included a plot when
$$ \cases{ a = (1,1,1)\\ b = (2,3,-2)\\ \vec t = (\frac{1}{\sqrt{2}},0,\frac{1}{\sqrt{2}}) } $$