You have to realize that when we are speaking of photons, we are speaking of elementary particles and their interactions are dominated by quantum mechanics, not classical mechanics, and in addition special relativity is necessary to calculate anything about them.
In general, we know about elementary particles because we observe their traces in detectors for almost a hundred years. We never see an electron, or a proton in the way we see a particle of dust.
This is the most visual detector, a bubble chamber photo of electromagnetic events.
Here we see some electromagnetic events such as pair creation or materialization of high energy photon into an electron-positron pair (green tracks), the Compton effect (red tracks), the emission of electromagnetic radiation by accelerating charges (violet tracks) (bremsstrahlung) and the knock-on electrons or delta ray (blue tracks)
Now lets see about your questions:
1) How did we arrive at "electrons exchange virtual photons and that's the cause of the electromagnetic force between them" from merely observing electrons absorbing or emitting photons?
That is not the way we arrived at this conclusion. A very large number of controlled scatterings, which is what this picture shows, of electrons on matter have been studied over the years and the theoretical framework of calculating the probability of the scatter and the angular distributions has been very well developed for years. This involves mathematics which cannot be handwaved. To start with, the crossection of an electron scattering on an electron can be written in a series of convoluted integrals which can be pictorially represented by Feynman diagrams. In those Feynman diagrams, the propagators of the interaction between the incoming and outgoing particles can be thought as virtual photons because they carry the quantum numbers of the photon but are off mass shell. So it is a convenient mathematical identification which defines virtual photons.
Anything between the incoming vertices and the outgoing vertices is virtual, and their reality depends on the correct representation of the quantum numbers for the exchanged particle, in this case photon quantum numbers.
2) If electrons throw photons at each other doesn't that mean that they should only scatter (repel)? If that is so why do magnets attract?
Virtual photons are not like balls, they are off mass shell, they are useful a mathematical construct .There is an interesting analog though where two boats throwing balls at each other represent the repulsive forces, and boomerangs the attractive.
Do these two phenomena happen here on earth naturally (no LHC or other particle accelerators), in the upper atmosphere, or only in the deep dark space?
There exist cosmic rays of all energies, the cosmic accelerator, and elementary particles were first seen in emulsions exposed to cosmic rays in high altitudes, for example the pion was thus discovered. So any process seen in accelerators can be found if looking hard enough in cosmic rays. Accelerators allow detail and exact measurements of crossections and branching ratios etc. because of the high statistics possible.
Best Answer
First of all, virtual particles are indeed a consequence of the uncertainty principle – without any quotation marks. Virtual particles are those that don't satisfy the correct dispersion relation $$ E = \sqrt{m^2 c^4 +p^2 c^2}$$ because they have a different value of energy by $\Delta E$. For such a "wrong" value of energy, they have to borrow (or lend) $\Delta E$ from the rest of the Universe. This is possible for a limited amount of time $\Delta t$ as long as the "negated" time-energy uncertainty relationship $$\Delta t \cdot \Delta E \leq \hbar / 2$$ is obeyed. One simply can't measure energy $E$ during too short an interval $\Delta t$ more accurately than with the error $\Delta E$ given by the formula above which makes it possible to borrow/lend this much energy for such a short time.
Pretty much by definition, virtual particles are effects that look like a temporary existence of a real particle which is bounded in time by the inequality above. The more virtual the particle is – the greater the deviation of the energy $\Delta E$ is – the shorter is the timescale over which the virtual particles may operate. In the limit $\Delta E\to 0$, the virtual particles become "real" which means that they may also be observed. For a nonzero value, they can't be observed and they're just "intermediate effects in between the measurements" that modify the behavior of other particles. Most explicitly, virtual particles appear as propagators (internal lines) of a Feynman diagram.
The electron is not necessarily "simulating" anyone, whatever "simulating" was supposed to mean. Instead, the electron may "emit" a virtual particle such as a photon. The emission of a real photon is impossible by the energy/momentum conservation: in the initial electron's rest frame, the energy is just $m_e c^2$ but it would get increased both by the extra kinetic energy of the final moving electron and by the positive photon's energy, thus violating the energy conservation law. But the electron may emit a virtual photon for which the energy conservation law is effectively violated (or the photon has a different energy, perhaps negative one, than it should have) which is OK for the time $\Delta t$ described above. As long as the photon disappears before this $\Delta t$ deadline arrives – it is absorbed by another charged particle, everything is fine and this intermediate history contributes to the probability amplitudes. That's why charged particles influence each other due to electromagnetism; this is how the virtual photons operate.
Concerning the last question, yes, virtual particles may interfere with the real ones. For example, if we study processes in an external electric field create by many coherent long-wavelength photons, there will still be Feynman diagrams with virtual photons in them. The amplitudes from these diagrams have to be added to the amplitudes with the real classical electric field, and only the result (sum) is squared in absolute value. That's what we mean by interference.
And yes, the effects of virtual particles on a isolated electron are equally likely in all directions and in this sense they "average out". An electron state with a sharply defined 3-momentum still remains an energy eigenstate and moves along a straight line. However, due to the constant emission and reabsorption of some virtual particles, the real electron-like energy eigenstate has a "cloud" of virtual photons around it. The symmetries of the theory such as the gauge symmetry and the Lorentz symmetry aren't broken by the virtual photons. After all, the virtual photons result from the theory whose Lagrangian does respect these symmetries and no anomaly breaks them.