The question is a slight extension of the question given here: Check whether a point is within a 3D Triangle
There is an elegant solution to this given by W. Heidrich, Journal of Graphics, GPU, and Game Tools,Volume 10, Issue 3, 2005.
Let $\vec{u}=P_2-P_1$, $\vec{v}=P_3-P_1$, $\vec{n}=\vec{u}\times\vec{v}$,
$\vec{w}=P-P_1$. We then have directly the barycentric coordinates of the projection $P'$ of $P$ onto $T$ as
- $\gamma=\left[(\vec{u}\times\vec{w})\cdot\vec{n}\right]/\vec{n}^2$
- $\beta=\left[(\vec{w}\times\vec{v})\cdot\vec{n}\right]/\vec{n}^2$
- $\alpha=1-\gamma-\beta$
The coordinates of the projected point is
- $P'=\alpha P_1+\beta P_2 +\gamma P_3$
The point $P'$ lies inside $T$ if
- $0\leq\alpha\leq 1$,
- $0\leq\beta\leq 1$, and
- $0\leq\gamma\leq 1$.
That algorithm is fine. But it's easier, once you've projected to a point $Q$ in the plane, to search in the plane. If $Q$ is interior to the triangle, you're done, as you observed.
Now you can test, for each edge, say $UV$, whether the projection $Q$ and the remaining point $W$ are on opposite sides of the line $UV$. If so, $W$ is not the closest point, and you can find the closest point by projecting onto $UV$: if the projection lies between $U$ and $V$, it's the closest point; if not, then one of $U$ or $V$ is.
Thomas Akenine-Moller has done a LOT of work on this kind of thing, and has many very clever algorithms for doing it with the smallest possible number of computations. To be more precise, he's studied ray-triangle interserction algorithms, but many of the ideas there apply here as well. Sample paper: Tomas Möller and Ben Trumbore. Fast, minimum storage ray-triangle intersection. J. Graph. Tools, 2(1):21–28, October 1997.
As for "have I missed anything", one way to think of it is that you start at the point $Q$ and draw ever-expanding disks until one touches the triangle. When you view it that way, it's pretty clear that the choices you described are exhaustive.
(Apologies for my earlier answer: I was just being too glib, and should have paused and thought for just a moment longer...)
Best Answer
You know that $$ \begin{align} P = \alpha A + \beta B + \gamma C &= \beta(B-A) + \gamma(C-A) + A\\ &= \alpha (A-B) + \gamma(C-B) + B \\ &= \alpha (A-C) + \beta(B-C) + C. \end{align} $$ We can classify the outside of the triangle in six regions:
1) $0\leq\beta$, $0\leq\gamma$, and $1\leq\beta+\gamma$, in which case the point is closest to $BC$. (In which case would it be closest to the $B$ or $C$?)
2) $0\leq\alpha$, $0\leq\gamma$, and $1\leq\alpha+\gamma$, in which case the point is closest to $AC$.
3) $0\leq\alpha$, $0\leq\beta$, and $1\leq\alpha+\beta$, in which case the point is closest to $AB$.
I'll let you work the other three cases which involve mixed signs, for example $1<\gamma$, $\beta<0$, for which the vertex $C$ is the closest point.