I don't know where you got your formula from, but but I derived it this way:
Field inside the solenoid$=\mu_0ni \hat{z}$ (say)
Since the material is ferromagnetic, there is an induced, bound surface current $K\hat{\phi}$ (and $K=M$, where $M$ is magnetization). The magnetization is uniform, so bound current is zero,$$
J_b~=~\nabla \times \left(M\hat{x}\right)~=~0
\,.$$
From the Lorentz force, $F=i \vec{l}\times \vec{B}$:
$$
\begin{align}
\Rightarrow F &= A\vec{K}\times\vec{B} \\
&=AK\left(\mu_0ni\right) \left(\hat{\phi}\times\hat{z}\right) \\
&=AM(\mu_0ni)\hat{r} \\
&=A(\mu_0ni)(\chi_m*H)\hat{r} \\
&=A(\mu_0ni)(\chi_m*\frac{\mu_0ni}{\mu_0})\hat{r} \\
&=\chi_mA\mu_0 \left(ni \right)^2\hat{r}\,,
\end{align}
$$where $A$ is area.
Here $A$ is the area of the surface that $K$ is flowing on, i.e., the curved surface of the cylinder$=2\pi RL$ where $R$ is radius and $L$ the length of ferromagnet. The force is radially out of the surface of the core, stretching it out as if to fill the coil.
I don't know why the force should depend on the "gap" between the two.
There is not really a simple answer to this question. It depends on the detailed material properties of the iron and the strength of the external magnetic field. What follows is a way to get an estimate of the force, but I would not be surprised if the results you get from this method are not terribly accurate.
When the iron is placed into a magnetic field, it gains a magnetic dipole moment $\vec{m}$ due to its magnetic susceptibility $\chi_m$. This is a property of the material of the ball, which tells you how strongly the atoms of the material respond to a magnetic field.
If the magnetic field is not too strong, and there are no hysteresis effects, then the resulting magnetic dipole moment of the ball is
$$
\vec{m} = \chi_m V \vec{B}
$$
where $V$ is the volume of the iron ball.
Finally, if the ball is sufficiently small (such that the strength of the coil's magnetic field does not vary substantially across it), then the force on the ball can be approximated as the force on a magnetic dipole:
$$
\vec{F} = - (\vec{m} \cdot \vec{\nabla}) \vec{B}
$$
which, after some mathematical manipulation, can be rewritten as
$$
\vec{F} = - \vec{\nabla} \left(\frac{1}{2} \chi_m V B^2 \right).
$$
So to calculate the force on the ball, you will need to know:
The volume of the ball. This is easy enough to figure out.
The magnetic susceptibility of the material of the ball. This is harder to determine; while the above-linked Wikipedia article gives a number for this, it is perhaps an over-simplification. To illustrate this, take a look at the table for the magnetic permeability of various materials; the permeability $\mu$ is related to the susceptibility by $\chi_m = \mu/\mu_0 - 1$. From this, we can see that permeability of the iron depends strongly on its purity, so this may be difficult to determine. (And I assume that when you say "iron" you mean "iron" rather than "steel"; steel is even harder to figure out.)
The magnetic field created by the coil as a function of position. This is also rather difficult to calculate from first principles, though it can be done (particularly if you're only interested in the case where the ball moves along the symmetry axis of the coil.) You could also in principle measure the magnetic field at several points along the desired path and interpolate between them to figure out the force. Also, remember that I said that the magnetic field had to be "not too strong" above; if the magnetic field is too strong, then the nice linear relationship between the magnetic dipole $\vec{m}$ and the magnetic field $\vec{B}$ breaks down.
Finally, note that the force will change as a function of position: closer to the magnet, $\vec{B}$ is larger and changes more rapidly with position, and so the ball will experience a greater force. This means that you'll have to use calculus to figure out the motion of the ball if that's the reason you want to find $\vec{F}$. But if you've gotten this far into this answer without getting lost, that shouldn't be terribly complicated for you.
Best Answer
This is not the Lorentz force, which describes electric and magnetic forces on electric charges, it is the force of a magnetic field on a magnetic dipole (here's the wiki page about it).
When the coil is activated the magnetic field magnetizes the ferromagnetic spheres in the same direction (this corresponds to the magnetic field aligning the electron spin dipoles in the atoms of the spheres). Now that they are magnetized the magnetic field applies a force to them and pulls them towards the coil.
Quantitatively, the magnetic field $\mathbf B$ induces a dipole moment $\mathbf m$ in the spheres which gives them a potential energy $$U = - \mathbf m\cdot\mathbf B$$
The corresponding force is $$\mathbf F = -\nabla U = \nabla(\mathbf m\cdot \mathbf B)$$ In words this (roughly) says the force pulls the spheres in the direction of increasing $\mathbf B$ field strength. I say roughly because that explanation ignores any increase of $\mathbf m$ of the spheres, but $\mathbf m$ will always be induced parallel to $\mathbf B$ so the overall effect remains the same: Since the magnetic field strength is larger closer to the coil, the force pulls the spheres towards the center of the coil. At the center the field strength is at a maximum magnitude which means $\mathbf m\cdot \mathbf B$ is at an extremum, i.e. $\nabla(\mathbf m\cdot \mathbf B) = 0$, which implies $\mathbf F = 0$.
A force of 0 at the center of the coil is why you see in the video that with the coil left on the spheres get stuck: Any displacement of a sphere from the center of the coil causes a force to pull it towards the center, so that the center is a stable equilibrium point of no net force.
Therefore in order for the coil to speed up the spheres as they pass, it has to be turned off before they reach the center, otherwise the field will pull them back in when they come out the other side.
The way it works on permanent magnets is similar, and has the same force equation $$\mathbf F = \nabla(\mathbf m\cdot \mathbf B)$$ Generally permanent magnets will have a much larger dipole moment than induced magnets, so the force is larger. However problem is that with permanent magnets, the dipole moment $\mathbf m$ is not induced in the direction of $\mathbf B$, rather it depends on the orientation of the permanent north and south poles of the sphere. The misalignment of $\mathbf m$ and $\mathbf B$ results in a torque in addition to the force $$\mathbf \tau = \mathbf m\times \mathbf B$$ which tries to keep the poles aligned with the field. This is bad for the accelerator: The spheres want to roll for efficient motion, but the field wants to keep their orientations fixed. This is probably what he meant in the description about permanent magnets having erratic behavior.