Let the helix be given by $(\cos t, \sin t, ht)$ (after scaling). If $P$ is your point $(a,b,c)$, and $Q = (\cos t, \sin t, ht)$ is the nearest point on the helix, then $PQ$ is perpendicular to the tangent at $Q$, which is just $(-\sin t, \cos t, h)$:
$-(\cos t - a)\sin t + (\sin t - b)\cos t + (ht - c)h = 0 $
This simplifies to $A \sin(t+B) + Ct + D = 0$ for some constants $A,B,C,D,$ as Moron said. But then you have to solve this numerically. There will be more than one solution in general, but (as Jonas Kibelbek pointed out in the comments) you only need to check the solutions with $z$-coordinate in the interval $[c-\pi h, c+\pi h)$.
First, vectors are considered a column vectors by default, so ${n_i}^T$ is a row vector. Then the denominator is the matrix product ${n_i}^Tn_i$, which is the same as the inner product $n_i\cdot n_i$, that is, it gives the square of the length of $n_i$:
$${n_i}^Tn_i=\|n_i\|^2\,.$$
Now introduce $m_i:=n_i/\|n_i\|$, this is already a unit vector ($\|m_i\|=1$).
We have
$$m_i\,m_i^T=\frac{n_i\,{n_i}^T}{\|n_i\|^2}=P_i\,.$$
We also have $P^2=m_im_i^T\cdot m_im_i^T=m_i\,\|m_i\|^2\, m_i^T=m_i\cdot1\cdot m_i^T=P_i$, and
${P_i}^T=(m_im_i^T)^T=((m_i)^T)^Tm_i^T=m_im_i^T=P_i$, so $P_i$ is indeed an orthogonal projection matrix.
Say, $v$ is parallel to the plane $i$, so that $v\perp n_i$, or, $v\perp m_i$. Then we have $m_i^Tv= m_i\cdot v=0$, so also $P_iv=0$. On the other hand $P_i(m_i)=m_i$. So, $P_i$ projects orthogonally to the line of $m_i$.
If you want the projection to the plane, then you have to take $I-P_i$ where $I$ is the identity matrix.
For (1), by the avove, we get that $P_i(p-o_i)$ is the orthogonal projection of the vector $p-o_i$ (this one goes from point $o_i$ to point $p$) to the line of $m_i$, so its length is indeed the distance from $p$ to the plane.
Alternatively, we can calculate it as
$$|(p-o_i)\cdot m_i|\,,$$
maybe it's a bit more simple..
Update:
For a linear transformation $T:V\to V$ to be idempotent (i.e. $T^2=T$), is equivalent to that $T$ is a projection, but not necessarily orthogonal, i.e. there are subspaces $U,W$ such that $T$ is the projection onto $U$ along $W$. Here $U$ and $W$ are disjoint: $U\cap W=\{0\}$, and together span the whole $V$, i.e. every vector $v$ can be uniquely written in the form $v=u+w$ for $u\in U,\,w\in W$. [This fact is denoted by $V=U\oplus W$.]
Now, if $T^2=T$, then let $U:={\rm ran\,}T=\{Tv\mid v\in V\}$, and let $W:=\ker T=\{v\mid Tv=0\}$. Try to prove the statements above w.r.t $U$ and $W$, and that
$$T(u+w)=u$$
for all $u\in U$ and $w\in W$.
The additional condition that $T^*=T$ ensures in addition that $W\perp U$, so it is an orthogonal projection.
Best Answer
the line through $(x_1, y_1, z_1)$ and orthogonal to the plane $ax + by + cz = d$ has the parametric representation $$x = x_1 - at, y = y_1 - bt, z = z_1 - ct$$ where $t$ is a real number. for this point to be on the plane $t$ needs to satisfy
$$t_1 = {ax_1+by_1 + cz_1 - d \over a^2 + b^2 + c^2}$$ and the shortest distance is $$\sqrt{a^2 + b^2 + c^2} |t_1| = {|\ ax_1+by_1 + cz_1 - d\ |\over \sqrt{a^2 + b^2 + c^2}} $$