It is indeed correct that only the difference between two potential energies is physically meaningful. An in-depth explanation follows. For the rest of this answer, forget everything you know about potential energy.
I suppose you know that when you have a conservative force $\vec{F}$ acting on an object to move it from an initial point $\vec{x}_i$ to a final point $\vec{x}_f$, the integral $\int_{\vec{x}_i}^{\vec{x}_f}\vec{F}\cdot\mathrm{d}\vec{s}$ depends only on the endpoints $\vec{x}_i$ and $\vec{x}_f$, not on the path. So imagine doing this procedure:
- Pick some particular starting point $\vec{x}_0$
Define a function $U(\vec{x})$ for any point $\vec{x}$ by the equation
$$U(\vec{x}) \equiv -\int_{\vec{x}_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
This function $U(\vec{x})$ is the definition of the potential energy - relative to $\vec{x}_0$. It's very important to remember that the potential energy function $U$ depends on that starting point $\vec{x}_0$.
Note that the potential energy function necessarily satisfies $U(\vec{x}_0) = 0$. So you can write
$$U(\vec{x}) - U(\vec{x}_0) = -\int_{\vec{x}_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
Now why would you do that? Well, suppose you choose a different starting point, say $\vec{x}'_0$, and define a different potential energy function
$$U'(\vec{x}) \equiv -\int_{\vec{x}'_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
(Here I'm using the prime to indicate the different choice of reference point.) Just like the original potential energy function, this one is equal to zero at the starting point, $U'(\vec{x}'_0) = 0$. So you can also write this one as a difference,
$$U'(\vec{x}) - U'(\vec{x}'_0) = -\int_{\vec{x}'_0}^{\vec{x}} \vec{F}\cdot\mathrm{d}\vec{s}$$
The neat thing about this definition is that even though the potential energy itself depends on the starting point,
$$U(\vec{x}) \neq U'(\vec{x})$$
the difference does not:
$$U(\vec{x}_1) - U(\vec{x}_2) = U'(\vec{x}_1) - U'(\vec{x}_2)$$
Check this yourself by plugging in the integrals. You'll notice that anything depending on the starting point cancels out; it's completely irrelevant.
This is good because the choice of the starting point is not physically meaningful. There's no particular reason to choose one point over another as the starting point, just as if you're on a hilly landscape, there's no particular reason to choose any one level to be zero height. And that's why potential energy itself is not physically meaningful; only the difference is.
Now, there is a convention in (very) common use in physics which says that when possible, unless specified otherwise, the starting point is at infinity. This allows you to get away without saying "difference of potential energy" and explicitly defining a starting point every time. So when you see some formula for potential energy, like
$$U(\vec{r}) = -\frac{k q_1 q_2}{r}$$
unless specified otherwise it is actually a difference in potential energy relative to infinity. That is, you should read it like this:
$$U(\vec{r}) - U(\infty) = -\frac{k q_1 q_2}{r}$$
Note that the function $\frac{k q_1 q_2}{r}$ goes to zero as $r\to\infty$. That's not a coincidence. It was chosen that way to ensure that $U(\infty) = 0$, so that you could insert it the same way I inserted $U(\vec{x}_0)$ in the calculations above. (This is just another way of saying it was chosen to make the $\frac{1}{a}$ term in the integral you did go away, so you don't have to write it.)
There are some situations in which you can't choose the reference point to be at infinity. For example, a point charge with an infinite charged wire has an electrical potential energy of
$$U = -2kq\lambda\ln\frac{r}{r_0}$$
where $r$ is the distance between the point charge and the wire. This potential energy function decreases without bound as you go to infinite distance ($r\to\infty$), it doesn't converge to zero, so you can't use infinity as your starting point. Instead you have to pick some point at a finite distance from the wire to be your starting point. The distance of that point from the wire goes into that formula in place of $r_0$.
By the way, electrical potential (not potential energy) is something a little different: it's just the potential energy per unit charge of the test particle. For a given test particle, it's proportional to electrical potential energy. So everything I've said about applies equally well to electrical potential.
Well actually, even if the general integral formula does not work since the charge is infinitely extended as correctly pointed out in the other answer, there is a way out using a suitable cutoff and a, say, renormalization procedure. Integrate from $-L$ to $L$ instead of from $-\infty$ to $+\infty$.
\begin{align}
V(\mathbf r) &=\frac{1}{4\pi\epsilon_0} \int_{[-L,+L]} \frac{\rho'_l}{\lvert \mathbf r - \mathbf{r'} \rvert}\mathrm{d}l' \\
&=\frac{\rho_l}{4\pi\epsilon_0} \int_{-L}^{+L}\frac{1}{\sqrt{(r')^2+(z-z')^2}}\mathrm{d}z'\\
&=\frac{1}{4\pi\epsilon_0} \left[ \ln \left|z-z'+\sqrt{(r')^2+(z-z')^2}\right|\right]^{+L}_{-L}\\
&= -\frac{\rho_l}{4\pi\epsilon_0} \ln \frac{r^2}{2L^2} + O(1/L)
\end{align}
The result, using $\ln(a^n) = n \ln a$ and $\ln(a/b)= \ln a - \ln b$ can be re-written as
$$V(r)= -\frac{\rho_l}{2\pi\epsilon_0} \ln \frac{r}{\ell} + O(\ln (L/\ell))$$
for an arbitrarily fixed length unit $\ell$. We now renormalize the result just by dropping the final divergent term $ O(\ln(L/\ell))$ obtaining
$$V(r)= -\frac{\rho_l}{2\pi\epsilon_0} \ln \frac{r}{\ell}$$
where you see that the so-called finite-renormalization ambiguities are all encapsulated in the arbitrary length scale $\ell$. However
the gradient of the found function is not affected by the arbitrary choice of $\ell$.
If you compute (minus) the gradient of the found $V$ you have the correct electric field also obtained by Gauss law and symmetry arguments:
$$\mathbf{E}(\mathbf{r})=\frac{\rho_l}{2\pi\epsilon_0 r}\hat{\mathbf{r}}\:.\tag{1}$$
There is a physical interpretation of the outlined procedure. You may consider a class of charged segments of length $L_n$ with $L_n \to + \infty$ as $n\to +\infty$ and compute $V_{L_n}({\bf r})$ for every $L_n$ where ${\bf r}$ is a fixed point in space. The crucial observation is that, since the potential is always defined up to an additive constant, at each step you can re-define $V_{L_n}$ subtracting a constant $C_n$ which logarithmically diverges as $L_n\to +\infty$. This limit procedure leads to the result found above.
There is no way to avoid the introduction of the arbitrary scale $\ell$ because the functional form of the electric field (1) says that any potential function producing it as gradient must have a logarithmic form. However the problem is that the argument of the logarithm is $r$ which has the dimension of a length. So, to make this argument dimensionless, one is forced to fix some scale $\ell$ arbitrarily which, however does not affect the electric field as it disappear when computing the derivative.
Best Answer
No, it's okay. The pontential difference increases as you go farther. The less you move away, the more similar potential you have (little difference).
By the way
$$\Delta V = -\dfrac{\lambda}{2\pi\varepsilon_0} \ln \left(\frac{r_F}{r_o}\right)$$
and the voltage difference increases when you go further, but in a negative sense, which means it becomes "more negative" as you move away.