I think the textbook description is a case of getting out of hand.
If you check http://en.wikipedia.org/wiki/Drift_velocity it says the the drift velocity is a factor of the applied electric field and the carrier mobility. However if you research carrier mobility, http://en.wikipedia.org/wiki/Electron_mobility says that the kind of acceleration you describe is possible in solids for distances/times as small/short as the mean free path/time but that in those cases "drift velocity and mobility are not meaningful." So I think drift velocity is supposed to be for multiple scatterings. (You even have one drift velocity for holes and another for electrons.)
To be fair to the textbook presentation, such a description usually comes up in a discussion of why many materials are Ohmic, specifically the situation of low-field mobility, which is often constant (drift velocity being proportional the applied electric field). The idea being that there is a very large thermal speed of the carrier, and that for the applied electric field the velocity didn't change all that much percentage wise. So for an effective characteristic time between effective collisions, the time between collisions didn't really change much. So stronger fields just have proportionally stronger effects and changes in velocity
In that sense the factor of two isn't the issue, it's just a characteristic time, and in the low field limit that characteristic time doesn't change for different applied fields. It's not literally a time between literal classical collisions of classical particles.
What is the characteristic time?
It is important to make sure you know that the average velocity you are trying to find (the drift velocity) isn't really the time averaged velocity of any particular electron. Instead, what you do is you take the momentum of every electron and add them up to get the total momentum of the electrons, divide it by the number of electrons to get the average momentum of the electrons, and then solve that for velocity. So it's really a spatial average. There are senses where that can be very close to a time average, but that could be less than helpful.
OK, so some electrons are moving slower, some are moving faster. At any moment (or small time interval) some of them are getting close enough to the other parts of the wire (or close enough to each other) that they exchange momentum with them so don't keep all of the momentum the electric field would give them at that momentum (or small time interval). In absence of an applied field, the electrons as a group have a distribution of velocities, some slower, and some faster, and some pointing in different directions. In a time interval, it might be that the electron in a particular region changes from a faster one into a slower one, or from one going in one direction to one going in another direction. But the distribution of velocities stays the same.
So in reality, the electric field can supply momentum to each electron, but sometimes instead of getting that momentum and going more in that direction, the electron exchanges it with others.
Now that we know what really happens, lets look at the case of no applied electric field, but make a massively gross oversimplification. This massively gross oversimplification is to say that the electron goes in a straight line at whatever velocity is has and then gets smacked really quickly and starts going in a random direction and random speed (but randomly chosen from a probability distribution like that of the whole collection of electrons). That massive and gross oversimplification correctly gets that the velocity of an electron is sometimes unchanging, and sometimes changes, but that overall has the probability distribution that the collection has. (Which depends on temperature, for normal metals and room temperature, most of the electrons are travelling at near $10^6$ meters per second, close to 1% of the speed of light, pretty fast, and not many are going much faster and not many are going much slower and they are equally travelling in all directions within the wire.)
So you can oversimplify and then try to oversimplify how long between collisions, which is roughly related to how far apart things are and your speed. Your speed is mostly the same. So there is a time between collisions. That is the characteristic time, but there isn't really a drift velocity there because it isn't really moving in straight lines then getting whacked really hard really quickly. It's just a story that is equivalent enough to get some answers right. Enough of the features are correct to explain why a material is Ohmic in the sense that for lots of different fields, the proportionality between current density and electric field is constant. It isn't really constant, it depends on density and temperature and such.
So what is drift velocity really and where does it come from really?
The what is actually correct, it is the spatial average, so you have huge velocity vectors ($10^6$ meters per second) belonging to a much huger number of electrons ($10^{20}$ or more) pointing in many directions pointing every which way. So the $10^{20}$ plus vectors are pointing in $10^{20}$ or more different directions. But they don't average out to zero when there is an applied field, there are a bit more pointing one way than the opposite way, and a bit larger in some directions than others. That spatial average of the velocities is the drift velocity. It's really just the total current density written as if were an average velocity of the charges. It comes about because of the net effect of the wire and the electric field. Let's talk about that more.
How does the scattering counter the electric field?
Forget the electric field for now, let's see what to wire does and can do to electrons. The wire has mobile carriers that can move about and it was parts that are more or less fixed in location relative to each other. Like parts of train, the whole train can move, but each chair in the train has a fixed distance from the other chairs of the train.
If the train starts to move and a person is sitting on the chair, the chair can push them until they are moving with the train. Same with the wire. You can place your hands along every part of the wire and move it. When you do, the whole wire moves just like the train moves. But in the train, if there was a ball tossed straight up in the air right before the train started moving then for a bit you have a ball not moving in a moving train. Same with the mobile electrons, in some sense it is free from the wire, it isn't stuck to any one place, so the wire starts to move and the electron finds itself at rest (not really since most of the time it is moving at $10^6$ meters per second, but on spatial average it is at rest). So the electrons find themselves to be at rest (spatially averaged) in a moving wire. But just like the ball eventually hits the back of the train or the floor of the train, the electrons do get dragged along eventually. The wire rushing towards them hits it harder from one direction than the other. At first they are running around in all directions at equal speeds but when they hit part of the wire that part of the wire is moving so when they hit head on they get pushed harder and when they overtake it they get pushed back less hard, the net effect is that they start to move in the direction of the wire, at the speed of the wire.
So now let's bring up the electric field. Imagine an electric field pointing in the x direction, then it wants to accelerate electrons in the -x direction. But what if the wire was moving in the x direction. If it moved at the right speed it would make the electrons move in the x direction exactly as much on spatial average as the electric field makes them go in the -x direction. So the net effect is the electrons would move around equally in all direction.
That is exactly what happens in the frame moving at drift velocity. In that frame the electrons move equally in all direction, the wire is moving at drift velocity and there is an electric field. That is literally where the drift velocity come from. The speed of the wire relative to the average velocity of the electrons that produces as much acceleration from electron-wire interactions as the the electron gets from electron-electric-field interactions.
Best Answer
It could be instructive to work out yourself the ac response of the Drude model or its simplified version (see my own posts here and here). The key point is that electron acceleration by the electric field is already taken into account in the electron mobility/conductance $$ \sigma_0 = \frac{nq^2\tau}{m}, $$
since the electrons are constantly reaccelerated by the field after being scattered from the lattice (impurities and phonons). In particular, the ac conductance is predicted to be $$ \sigma(\omega)=\frac{\sigma_0}{1-i\omega\tau}= \frac{\sigma_0}{1+\omega^2\tau^2} + \frac{i\omega\sigma_0}{1+\omega^2\tau^2} $$ As one can see, the conductance has an imaginary part, characterizing the current retardation in respect to the electric field. Moreover, it decreases in magnitude at higher frequencies - that is, the cvurrent decreases with increasing the frequency, as $$ \mathbf{j}(\omega)=\sigma(\omega)\mathbf{E}(\omega) $$