Assume that both the surface and the bulk are insulators with vacuum permittivity $\varepsilon_0$, so that the charges cannot redistribute themselves.
Consider first the electric field
$$\vec{E}~=~\frac{\sigma}{2\varepsilon_0}
\begin{pmatrix} {\rm sgn}(x)\\0\\0 \end{pmatrix} $$
associated with a uniformly charged capacitor plate at $x=0$ parallel to the $yz$ plane.
Consider next the electric field
$$\vec{E}~=~\frac{\sigma}{2\varepsilon_0}
\begin{pmatrix} 0\\{\rm sgn}(y)\\0 \end{pmatrix} $$
associated with a uniformly charged capacitor plate at $y=0$ parallel to the $xz$ plane.
Now construct a simple counterexample (where $\vec{E}$ is not perpendicular to the surface) by adding together the charge distributions in situation 1 and 2, cf. figure. Use superposition principle to determine
$$\vec{E}~=~\frac{\sigma}{2\varepsilon_0}
\begin{pmatrix} {\rm sgn}(x)\\{\rm sgn}(y)\\0 \end{pmatrix}, $$
Figure:
Capacitor plate
|
\ | / E-field lines
\ \ \|/ / /
\ \ | / /
\ \ | / /
----------------------------- Capacitor plate
/ / | \ \
/ / | \ \
/ / /|\ \ \
/ | \
|
The answer by @BrianMoths is correct. It's worth learning the language used therein to help with your future studies. But as a primer, here's a simplified explanation.
Start with your charge distribution and a "guess" for the direction of the electric field.
As you can see, I made the guess have a component upward. We'll see shortly why this leads to a contradiction.
Now do a "symmetry operation," which is a fancy phrase for "do something that leaves something else unchanged. In this case, I'm going to reflect everything about a horizontal line. I mean everything.
The "top" of the sheet became the "bottom." This is just arbitrary labeling so you can tell I flipped the charge distribution. The electric field is flipped too. (Imagine looking at everything in a mirror, and you'll realize why things are flipped the way they are.)
Hopefully, everything is okay so far. But now compare the original situation with the new inverted one.
You have exactly the same charge distribution. You can't tell that I flipped it, except for my arbitrary labeling. But if you have the same charge distribution, you ought to also have the same electric field. As you can see, this is not the case, which means I made a mistake somewhere.
The only direction for the electric field that does not lead to this contradiction is perpendicular to the sheet of charge.
Best Answer
Yes, a discontinuity in the electric field is always associated with a charge distribution with infinite density. That's a necessary implication of Gauss' law. To think of it qualitatively, where there is a discontinuity in the field, there must be field lines either starting or ending. (If it's not obvious why that is, think about the fact that the electric field magnitude is different on one side of the discontinuity than on the other. There will be more field lines on the side with the stronger side, or else lines pointing in different directions on the different sides if the amplitude is positive on one side and negative on the other.) But electric field lines start and end only on charges, or at infinity.
To think about it even more physically, consider the force on a positive test charge. It will be different on either side of the discontinuity. But electric forces are exerted by charges, so the only way to produce a jump in the force across the discontinuity is to have some charge there.
In fact, it turns out that if the electric field amplitude jumps by $\Delta E$ across a discontinuity, there must be a surface charge there (charge per unit area) of magnitude $\epsilon_\circ \Delta E$.