I've never seen an authoritative explanation for the choice of the lower case letter $\ell$ or $l$ to denote an arbitrary prime different from a given prime $p$. This now has its own LaTeX command \ell, but has been in use at least since the old work of Taniyama and Weil involving L functions. That use of the upper case letter might have suggested the lower case here, I guess(?) The letter q would seem more natural in elementary number theory. The write-up of Serre's 1967 McGill lectures was published in 1968 by W.A. Benjamin under the title Abelian l-adic representations and elliptic curves. There his convention is to denote prime numbers by $\ell, \ell', p, \dots$, stating: "we mostly use the letter $\ell$ for $\ell$-adic representations and the letter $p$ for the residue characteristic of some valuation".
I've heard this question raised but not answered quite a few times. For instance, after a colloquium talk in Hamburg given by Bhama Srinivasan on Deligne-Lusztig characters, the elderly Ernst Witt asked the non-technical question I've just raised. (He had done impressive work in his youth but became a convert to the Nazi cause without apparently committing any war crimes. Possibly he was the young man reported to have shown up once at Emmy Noether's seminar wearing a pro-Nazi uniform. In old age he had retained some mental acuity but developed phobias about for example the flooring material in the math tower, which required talks like the ones Bhama and I gave to move to a remote building.)
[ADDED] Both Franz and quim point in the direction of how the symbol $l$ became common for prime numbers in Hilbert's development of Kummer's work. There he considers an $l$th root of unity ($l$ an odd prime) instead of $\lambda$ used earlier by Kummer. Later on I guess it became a default option for many people to use $l$ for a prime different from a given prime $p$, especially when $q$ became used commonly for a power of $p$.
Best Answer
This elaborates quim's answer. Kummer did indeed use $\lambda$ for denoting primes (in connection with cyclotomic fields); he borrowed the notation from Jacobi's articles on cyclotomy as well as from his notes of the number theory lectures in 1836/37. When Hilbert rewrote Kummer's contributions in his Zahlbericht, he started the chapter on cyclotomic fields with "Let $l$ denote an odd prime number". The reason for switching from the Greek to the Latin alphabet was Hilbert's custom to use Latin letters for rational numbers. Hilbert also used ${\mathfrak l}$ for prime ideals above $2$.
Edit. For what it's worth: Euler used primes $\lambda n + 1$ in art. 92 of his article E449.