It is indeed correct that only the difference between two potential energies is physically meaningful. An in-depth explanation follows. For the rest of this answer, forget everything you know about potential energy.
I suppose you know that when you have a conservative force $\vec{F}$ acting on an object to move it from an initial point $\vec{x}_i$ to a final point $\vec{x}_f$, the integral $\int_{\vec{x}_i}^{\vec{x}_f}\vec{F}\cdot\mathrm{d}\vec{s}$ depends only on the endpoints $\vec{x}_i$ and $\vec{x}_f$, not on the path. So imagine doing this procedure:
- Pick some particular starting point $\vec{x}_0$
Define a function $U(\vec{x})$ for any point $\vec{x}$ by the equation
$$U(\vec{x}) \equiv -\int_{\vec{x}_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
This function $U(\vec{x})$ is the definition of the potential energy - relative to $\vec{x}_0$. It's very important to remember that the potential energy function $U$ depends on that starting point $\vec{x}_0$.
Note that the potential energy function necessarily satisfies $U(\vec{x}_0) = 0$. So you can write
$$U(\vec{x}) - U(\vec{x}_0) = -\int_{\vec{x}_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
Now why would you do that? Well, suppose you choose a different starting point, say $\vec{x}'_0$, and define a different potential energy function
$$U'(\vec{x}) \equiv -\int_{\vec{x}'_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
(Here I'm using the prime to indicate the different choice of reference point.) Just like the original potential energy function, this one is equal to zero at the starting point, $U'(\vec{x}'_0) = 0$. So you can also write this one as a difference,
$$U'(\vec{x}) - U'(\vec{x}'_0) = -\int_{\vec{x}'_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
The neat thing about this definition is that even though the potential energy itself depends on the starting point,
$$U(\vec{x}) \neq U'(\vec{x})$$
the difference does not:
$$U(\vec{x}_1) - U(\vec{x}_2) = U'(\vec{x}_1) - U'(\vec{x}_2)$$
Check this yourself by plugging in the integrals. You'll notice that anything depending on the starting point cancels out; it's completely irrelevant.
This is good because the choice of the starting point is not physically meaningful. There's no particular reason to choose one point over another as the starting point, just as if you're on a hilly landscape, there's no particular reason to choose any one level to be zero height. And that's why potential energy itself is not physically meaningful; only the difference is.
Now, there is a convention in (very) common use in physics which says that when possible, unless specified otherwise, the starting point is at infinity. This allows you to get away without saying "difference of potential energy" and explicitly defining a starting point every time. So when you see some formula for potential energy, like
$$U(\vec{r}) = -\frac{k q_1 q_2}{r}$$
unless specified otherwise it is actually a difference in potential energy relative to infinity. That is, you should read it like this:
$$U(\vec{r}) - U(\infty) = -\frac{k q_1 q_2}{r}$$
Note that the function $\frac{k q_1 q_2}{r}$ goes to zero as $r\to\infty$. That's not a coincidence. It was chosen that way to ensure that $U(\infty) = 0$, so that you could insert it the same way I inserted $U(\vec{x}_0)$ in the calculations above. (This is just another way of saying it was chosen to make the $\frac{1}{a}$ term in the integral you did go away, so you don't have to write it.)
There are some situations in which you can't choose the reference point to be at infinity. For example, a point charge with an infinite charged wire has an electrical potential energy of
$$U = -2kq\lambda\ln\frac{r}{r_0}$$
where $r$ is the distance between the point charge and the wire. This potential energy function decreases without bound as you go to infinite distance ($r\to\infty$), it doesn't converge to zero, so you can't use infinity as your starting point. Instead you have to pick some point at a finite distance from the wire to be your starting point. The distance of that point from the wire goes into that formula in place of $r_0$.
By the way, electrical potential (not potential energy) is something a little different: it's just the potential energy per unit charge of the test particle. For a given test particle, it's proportional to electrical potential energy. So everything I've said about applies equally well to electrical potential.
In physics voltage is usually defined as potential difference between two points of a circuit:
$$V_{ab}=\phi_a-\phi_b.$$
In other words:
- potential is a potential energy per unit charge measured in respect to an agreed reference point (e.g., an infinitely remote point), i.e. work done when bringing the charge from this point
- voltage is the work done when moving a charge between the two specified points.
One typically has the same reference point for all the potentials, but voltages can be defined between any two points.
Finally, potential is an extremely general concept, used to describe electric and electromagnetic phenomena well beyond circuit theory, whereas voltage is a term mainly limited to specific applications.
Update
The question has been further discussed in the comments, and it was agreed that much of my answer indeed reiterates the information already given in the OP. I therefore state below the main point, as it came out of the discussion:
One can say that potential is a voltage between the point of interest and the reference point. I think the real difference is indeed the usage: one will rarely speak of voltage in atomic or nuclear physics, but in case of semicodnuctor nanostructures the term is rather used. Some may say that potential is "more scientific term", as opposed to engineering.
Best Answer
The concepts of potential and potential energy are both used for both electric fields and gravitational fields. In light of some of the existing answers to your question, I think we all now understand that. The remaining question is why we talk about the potential more often when discussing electricity and the potential energy more often when discussing gravity.
In my opinion this is a habit formed by many of us because of the nature of the basic problems and early experiments in the respective areas.
The "discovery" of gravity was in the context of falling objects and orbiting planets, which is also the context of most introductory problems involving gravity. In those contexts, one is discussing a specific, identifiable object within the field, so it is natural to consider the potential energy relevant to that object.
For electricity, the topics early in its study are about currents and batteries where one is discussing a flow of many charged particles, no one of which is particularly important (or visible). Even when studying static electricity with charged metal objects, the charge often cannot be treated as a single entity, since it moves about within the object to equalize the internal potential. So we mostly talk about potential because it is the more convenient concept.