Suppose that $E = (x_1, y_1, z_1)$ is a point and $P$ is a plane described by the equation $A(x-x_0)+b(y-y_0) +C(z-z_0)= Ax+By+Cz+D = 0$.
Since the distance from $E$ to $P$ is the length of the projection of $\vec{RE}$ onto the normal vector $\vec{n}$ and by the formula of projection, I think we have
$$\text{distance} = \frac{\vec{RE}\cdot \vec{n}}{||\vec{n}||^2} = \frac{\vec{v}\cdot \vec{n}}{||\vec{n}||^2}.$$
Why does the distance from $E$ to $P$ equal
$$\frac{|Ax_1 + By_1 + Cz_1 + D|}{\sqrt{A^2+B^2+C^2}} = \frac{\vec{v}\cdot \vec{n}}{||\vec{n}||}$$
instead of
$$\frac{|Ax_1 + By_1 + Cz_1 + D|}{A^2+B^2+C^2}= \frac{\vec{v}\cdot \vec{n}}{||\vec{n}||^2}?$$
Best Answer
Your mistake is in the formula $$\text{distance} = \frac{\vec{RE}\cdot \vec{n}}{||\vec{n}||^2}$$
The correct formula is $$\text{distance} = \frac{\vec{RE}\cdot \vec{n}}{||\vec{n}||}.$$
which gives you the correct result $$\frac{|Ax_1 + By_1 + Cz_1 + D|}{\sqrt{A^2+B^2+C^2}}$$
Note that $$\text{distance} =||\vec{RE}||\cos(\theta)= \frac{\vec{RE}\cdot \vec{n}}{||\vec{n}||}$$.