Suppose we are given two triangles $ABC$ and $DEF$. We can assume nothing about them other than that they are in the same plane. The triangles may or may not overlap. I want to algorithmically determine the area (possibly $0$) of their overlap; call it $T_{common}$.
We have a multitude of ways of determining the areas of $ABC$ and $DEF$; among the "nicest" are the Heronian formula, which is in terms of the side lengths alone, and
$T = \frac{1}{2} \left| \det\begin{pmatrix}x_A & x_B & x_C \\ y_A & y_B & y_C \\ 1 & 1 & 1\end{pmatrix} \right| = \frac{1}{2} \big| x_A y_B – x_A y_C + x_B y_C – x_B y_A + x_C y_A – x_C y_B \big|$
which is in terms of the coordinates alone.
Obviously, there does exist a function from $A,B,C,D,E,F$ to $T_{common}$, but my question is: is there a "nice" (or even not-"nice") expression for $T_{common}$ in terms of the $x$ and $y$ coordinates of $A,B,C,D,E,F$?
I've drawn out on paper what I think are the various cases, but my issues with this approach are: identifying the case is a job in itself, which I can't easily see how to algorithmise ("just look at a picture" doesn't work for a computer); even within each case the algebra is fiddly and error-prone; and I have little confidence that I've enumerated all possible cases and got the computations right!
In my imagination there is a neat approach using ideas from analysis (treating the triangles as functions from $\mathbb{R}^2$ to $\{0,1\}$ and… multiplying them??) but I have no idea whether that's just a flight of fancy or something workable.
Best Answer
Here's an approach that doesn't require any ingenuity or case analysis. Use the Sutherland-Hodgman algorithm to clip one triangle against the other, yielding a (possibly degenerate) polygon representing the region of overlap, then compute its area.