The answer is not in general. There exist pathological charge densities for which the electric potential has discontinuities. Fortunately, these can always be ruled out on physical grounds: they usually have finite amounts of opposite charge infinitely close to each other, which allows for arbitrarily high electric fields, and therefore divergent energy densities and infinite configuration energies.
How does this work? Take two conducting plates and put their edges in contact. I'm quite fond of using the north and south hemispheres of a given sphere, separated at the equator (which has the advantage of not introducing any extraneous infinities), but for simplicity let's use two semi-infinite planes, in the $x,z$ plane and separated by the $x$ axis.
Now, connect both plates to a voltage source and set one to a potential $+V_0$ and the other to $-V_0$. The potential at the $x$ axis is (of course!) discontinuous, by construction.
If that makes you uneasy, consider spreading uniform but opposite surface charges $\pm \sigma_0$ on the $\pm z$ plates. Fix some positive coordinate $z_0$. Then if you zoom in more and more towards the plates (i.e. as you take $x\rightarrow 0$) the plates loom larger and larger, and when $x\ll z_0$ the $-$ plate looks so far away that the local potential has to be positive, and bounded away from zero. Similarly, if $z_0<0$ the potential close enough to the plates must be negative. Thus you have points arbitrarily near the $x$ axis which have potentials bounded away from zero, both positive and negative, so the potential is discontinuous there.
So why is this not a problem? In these pathological examples you are putting a lot of negative charge, in conducting surfaces, next to a lot of negative charge. This assumes that you have separated them with an insulator of zero thickness which can withstand large (albeit finite) potential differences (so infinite electric fields). This is just not physical: any dielectric will break down or spark as you approach these conditions. As soon as you introduce a finite separation, the discontinuity goes away.
More generally the electrostatic potential can have discontinuities. These are most usually considered as surface discontinuities, in which case they are known as dipole layers, which you can think of as infinitely thin capacitors set at constant potential. (Therefore, they must have two layers of opposite but infinite charge, which is again not physical.)
A real dipole layer would be something like a lot of polar molecules, all pointing the same way normally to the surface. We know that the potential inside is not discontinuous and that the electric fields inside are large but finite, but if you're outside then all you see are two plane layers of charge which look for all the world like they are setting up a discontinuous potential, so you might as well put that into your math and assume it is infinitely thin.
Yes, a discontinuity in the electric field is always associated with a charge distribution with infinite density. That's a necessary implication of Gauss' law. To think of it qualitatively, where there is a discontinuity in the field, there must be field lines either starting or ending. (If it's not obvious why that is, think about the fact that the electric field magnitude is different on one side of the discontinuity than on the other. There will be more field lines on the side with the stronger side, or else lines pointing in different directions on the different sides if the amplitude is positive on one side and negative on the other.) But electric field lines start and end only on charges, or at infinity.
To think about it even more physically, consider the force on a positive test charge. It will be different on either side of the discontinuity. But electric forces are exerted by charges, so the only way to produce a jump in the force across the discontinuity is to have some charge there.
In fact, it turns out that if the electric field amplitude jumps by $\Delta E$ across a discontinuity, there must be a surface charge there (charge per unit area) of magnitude $\epsilon_\circ \Delta E$.
Best Answer
If you specify the potential as a function from real numbers to real numbers then the function is discontinuous at zero by virtue of the fact that it is not defined at zero. I.e., the definition of continuity $$ \lim_{r\to 0}V(r)\to V(0) $$ does not hold because V(0) is not defined (it is not a real number).
The analogy seems somewhat imperfect to me because both $E$ and $V$ go to infinity in the first example, but neither does in the second example. If E remains finite, phi will be continuous (as discussed below).
Typically $\phi$ is continuous since, for example, if $\phi$ had a discontinuity $\Delta\phi$ at some point $x_0$ then $$ \Delta\phi=\lim_{\delta\to 0}\phi(x_0+\delta)-\phi(x_0)=\lim_{\delta\to 0}\int_{x_0}^{x_0+\delta}\frac{d\phi}{dx}dx=\lim_{\delta\to 0}\int_{x_0}^{x_0+\delta}E_j(x)dx =\lim_{\delta\to 0}E_j(x_0)\delta\;, $$ which is zero unless E(x_0) blows up. I.e., for finite electric fields (even discontinuous finite electric fields) the potential remains continuous.
But, remember, in general, the potential does not have to be continuous everywhere (as evidenced by the case mentioned above of a point charge). This is because the governing equation for $\phi$ is $$ \nabla^2 \phi=-\rho/\epsilon_0\;, $$ where $\rho$ is the charge density. And, I'll bet that you can come up with some wacky charge densities (a single point charge is one case) where $\phi$ is discontinuous.