Given a set of $N$ points, the distance between one pair of points may not be the same as the distance between some other pair of points. I suppose you want to maximize the minimum of the set of distances between points, measured between all possible pairs of points.
Suppose the resulting distance is $2r$. This means you could draw a disk of radius $r$ around each of your points, and no pair of these disks would overlap (although some pairs of the disks would touch each other). You would then have found a packing of $N$ circles within a circle of radius $R + r$, where $R$ is the radius of your original circle.
If you have succeeded in finding the maximum value of the minimum distance between points, this packing of circles will be optimal.
Finding an optimal packing of $N$ circles within a larger circle is a difficult problem.
See http://en.wikipedia.org/wiki/Circle_packing_in_a_circle. There are many values of $N$ for which we aren't yet sure we have found an optimal packing; for example, the best known packing for $N = 14$ is only conjectured to be optimal.
In other words, there isn't a simple mathematical formula that will just give you the answer. In practice, you'll probably have to resort so some kind of brute-force method or evolutionary algorithm, and you will never be sure you've found the best possible answer, only that your last answer is better than any of the other ones you've found.
Minimizing the squared distance,
$$\begin{array}{ll} \text{minimize} & x^2 + y^2 + (z-c)^2\\ \text{subject to} & z^2 = x^2 + 4 y^2\end{array}$$
Let the Lagrangian be
$$\mathcal L (x,y,z,\lambda) := x^2 + y^2 + (z-c)^2 + \lambda (x^2 + 4 y^2 - z^2)$$
Taking the partial derivatives and finding where they vanish, we obtain
$$\begin{array}{rl} (1 + \lambda) \, x &= 0\\ (1 + 4\lambda) \, y &= 0\\ (1 - \lambda) \, z &= c\\ x^2 + 4 y^2 - z^2 &= 0\\\end{array}$$
We have two cases to consider.
$\color{blue}{\boxed{\lambda = -1}}$
In this case, $y = 0$ and $z = \frac c2$. The value of $x$ is given by
$$x = \pm \sqrt{ z^2 - 4 y^2 } = \pm \frac c2$$
Hence, we have the two points
$$(x,y,z) = \left( \pm \frac c2, 0, \frac c2 \right)$$
whose squared distance from $(0,0,c)$ is $\frac{c^2}{2}$.
$\color{blue}{\boxed{\lambda = - \frac 14}}$
In this case, $x = 0$ and $z = \frac{4c}{5}$. The value of $y$ is given by
$$y = \pm \frac 12 \sqrt{ z^2 - x^2 } = \pm \frac{2c}{5}$$
Hence, we have the two points
$$(x,y,z) = \left( 0, \pm \frac{2c}{5}, \frac{4c}{5} \right)$$
whose squared distance from $(0,0,c)$ is $\frac{c^2}{5} < \frac{c^2}{2}$. This is the minimal squared distance.
Example
Let $c = 5$. Hence, the points on the cone closest to $(0,0,5)$ are $(x,y,z) = ( 0, \pm 2, 4 )$. Here is a plot of the cone and the line segment whose endpoints are $(0,0,5)$ and $( 0, 2, 4)$
The length of the line segment is $\sqrt 5$.
Best Answer
You’ve got a good start. Letting the point be $\mathbf x = (x_1,\dots,x_n)$ and the equation of the plane $a_1y_1+\dots+a_ny_n+b=\mathbf a\cdot\mathbf y+b=0$, you can set up a Lagrange multiplier problem with objective function $f(\mathbf y) = \sum_{i=1}^n(y_i-x_i)^2$. Differentiating gives a system of linear equations of the form $$2\left(y_i-x_i\right)-a_i\lambda = 0,\tag{*}$$ so $$y_i-x_i = \frac12 a_i\lambda$$ and $$f(\mathbf y)=\frac14 \lambda^2 \sum_{i=1}^na_i^2,$$ therefore the minimal distance to the plane is $\frac12 |\lambda| \|\mathbf a\|$.
To find $\lambda$, add $-\frac12 a_i$ times each of the respective (*) equations to the equation of the plane, producing $$\frac12\lambda\sum_{i=1}^na_i^2+\sum_{i=1}^na_ix_i+b=0,$$ or $$\frac12\lambda\|\mathbf a\|^2 = -\mathbf a\cdot\mathbf x-b.$$ Combining this with the previously-derived expression for the distance to the plane gives $${|\mathbf a\cdot\mathbf x+b| \over \|\mathbf a\|}$$ for this distance, as required.