3dgeometry
Context:
A BoundingPolytope defines a polyhedral bounding region using the intersection of four or more half spaces. The region defined by a BoundingPolytope is always convex and must be closed.
Each plane in the BoundingPolytope specifies a half-space defined by the equation:Ax + By + Cz + D <= 0
source: from a javadoc, this is for programming purpose
I need the value of A, B, C, and D for a plane as in that equation and all I have is corner points of a rectangle that lies in that plane.
That sounds more or less sensible, but there are some difficult cases you've forgotten to check for. It also doesn't quite work when the rectangle isn't a square. I think the best solution is actually a little simpler.
For simplicity, let me give this rectangle some coordinates: call the bottom left point (0, 0), the bottom right point (a, 0), the top left point (0, b) and the top right point (a, b).
The embarrassingly garish image below shows that there are actually 8 regions you need to look at. The good news is: if k is outside the square, you can just read off the nearest point and calculate its distance directly. (For example, if k is in the light blue area above the square, with coordinates (x, y), the nearest point on the rectangle is (x, b). If it's in the yellow area to the left, with coordinates (x, y), the nearest point is (0, y). If it's in the pink area to the lower right, the nearest point is (a, 0).)
If k is inside the square, it's a little more annoying. But the basic recipe is the same. If k has coordinates (x, y), you need to work out which is smallest out of these four numbers: x, a-x, y, b-y. If x is the smallest, the nearest point is (0, y). If a-x is the smallest, the nearest point is (a, y). And so on.
There are also ambiguous points, like (a/2, b/2), which have more than one 'nearest point'.
I hope I haven't made a mistake here. My head is swimming with coordinates. Hopefully the idea is clear. :)
If $b,c,d,e$ are a rectangle and $a$ is coplanar with them, you need only check that $\langle b, c-b\rangle\le \langle a, c-b\rangle\le \langle c, c-b\rangle$ and $\langle b, e-b\rangle\le \langle a, e-b\rangle\le \langle e, e-b\rangle$ (where $\langle,\rangle$ denotes scalar product).
Best Answer
You only need $3$ of these corner points. Let these corner points be denoted $\vec{v_1}$, $\vec{v_2}$, and $\vec{v_3}$, treated as vectors from the origin to the point.
Then, the equation of your plane is: $$\left((x, y, z)-\vec{v_1}\right) \cdot \left[(\vec{v_3}-\vec{v_1})\times(\vec{v_2}-\vec{v_1})\right] = 0$$
Where $\times$ is the cross product, $\cdot$ is the dot product, and $(x, y, z)$ are the free variables describing the plane.