Protons and neutrons are referred to collectively as nucleons. Nucleons interact via the strong nuclear force, and this interaction can't be expressed by any simple equation. The reason is that nucleons are not fundamental particles. They're actually clusters of quarks.
Short range
The low-energy structure of nuclei is amazingly insensitive to the details of the nucleon-nucleon interaction that you pick as an approximation to the actual underlying quark-quark interaction. This is both good and bad. It's good because you don't need to understand very much about the nasty details in order to find out the properties of nuclei, e.g., why nuclei have the sizes and shapes they do. It's bad because it means that you can never infer very much about the interaction simply by observing properties of nuclei. As an example of how insensitive nuclear structure is to the details of the strong nuclear force, clusters of sodium atoms have magic numbers that match up with the first few magic numbers for nuclei; this is because these magic numbers only depend on the short-range nature of the interaction.
Other effects that can be understood based on the short range of the interaction are:
Nuclei act as though they have surface tension (so they resist being deformed).
Nuclei are most stable if they have even numbers of neutrons and even numbers of protons (because then the neutrons and protons can pair off in time-reversed orbits that maximize their spatial overlap).
Nucleons in an open shell tend to couple so as to form the minimum total angular momentum (the opposite of Hund's rules for electrons).
A residual interaction
The short-range nature of the nuclear interaction is very surprising, because the quark-quark force is believed to be roughly independent of distance. What's happening here is that nucleons are color-neutral, just as a hydrogen atom is charge-neutral. Just as hydrogen atoms "shouldn't" interact, nucleons "shouldn't" interact either. The forces between nucleons very nearly cancel out, and likewise the electrical forces between two neutral hydrogen atoms very nearly cancel out. The nonvanishing interaction comes from effects like the polarization of one particle by the other. For this reason, the nucleon-nucleon interaction is referred to as a residual interaction.
Other than its coupling constant and its range, what other features of the nuclear interaction are important for understanding low-energy nuclear structure?
Spin-orbit
There is a spin-orbit interaction, which is much stronger than, and in the opposite direction compared to, the one expected from special relativity alone.
Symmetry between neutrons and protons
The nuclear interaction remains unchanged when we transform neutrons to protons and protons to neutrons. For this reason light nuclei exhibit nearly identical properties if you swap their N and Z. Heavy nuclei don't have this symmetry, which is broken by the electrical interaction.
No qualitative features inferrable from sizes of nuclei
We do not get any clearcut, qualitative information about the interaction based on the observed sizes of nuclei. An extremely broad class of interactions between point particles results in n-body systems that have bound states and finite density. The finite density (i.e., the lack of a total collapse to a point) is essentially a generic result of the Heisenberg uncertainty principle. Only for certain special types of potentials that blow up to $-\infty$ at short ranges can one circumvent this (Lieb 1976).
A variety of models
Because the nucleon-nucleon interaction is a residual interaction, and nucleons are really composite rather than pointlike, the whole notion of a nucleon-nucleon interaction is an approximation, and one can model it in a variety of ways while still producing agreement with the data. In particular, some models have a hard, repulsive core, while others do not,(Chamel 2010, Stone 2006) and both types can reproduce the observed sizes of nuclei. This disproves the common misconception that such a hard core is needed in order to explain the sizes of nuclei.
References
Chamel and Pearson, 2010, "The Skyrme-Hartree-Fock-Bogoliubov method: its application to finite nuclei and neutron-star crusts," http://arxiv.org/abs/1001.5377
Lieb, Rev Mod Phys 48 (1976) 553, available at http://www.pas.rochester.edu/~rajeev/phy246/lieb.pdf
Stone and Reinhard, 2006, "The Skyrme Interaction in finite nuclei and nuclear matter," http://arxiv.org/abs/nucl-th/0607002
You answered to your own question, I believe.
The electrostatic force has an infinite range and it falls down with distance in an inverse square power law
$$ F_e \propto \frac{1}{r^2} $$
while the strong nuclear force behaves quite differently as it exhibits a property called confinement.
Confinement is still to be understood (it is related to one of the millennium prizes), but heuristically you can think of it as a force (for example acting on a pair of quark and anti-quark) that grows linearly with distance
$$F_s \propto r$$
just like a stretched spring. Although quite stronger than the electrostatic force, it happens that $r$ cannot stretch ad infinitum as the spring breaks into two pieces
and as such there is somehow a maximum distance after which the strong force does not attract any more.
As such nuclei, as you pointed out, will not be stable after some size as the electrostatic force will be the predominant force, which is repelling protons from each other.
Other references:
http://en.wikipedia.org/wiki/Strong_interaction
http://en.wikipedia.org/wiki/Millennium_Prize_Problems#Yang.E2.80.93Mills_existence_and_mass_gap
Best Answer
This question is not worded very well. To explain that, let us consider the simplest systems containing just two neutrons or two protons. These systems do not have bound states. That means that the reason two protons do not have a bound state is not because the protons repel each other electromagnetically (remember that strong forces are pretty much the same for protons and neutrons, so if there were a bound state for two protons, there would also be a bound state for two neutrons), but because the strong force is spin-dependent, and two protons or two neutrons cannot have the same spin projections because of the Pauli principle, and the strong force between two nucleons with the opposite projections of spins does not enable a bound state. Remember that the system containing one proton and one neutron does have a bound state (deuteron) with the same spin projections for the neutron and the proton, as these two particles are distinguishable, and the Pauli principle does not extend to their system. Let me just add that at the distance of about 1 fermi characteristic for the strong force the electromagnetic energy for two nucleons can be orders of magnitude smaller than the energy of their strong interaction.