The drift velocity does not depend on the length or the cross sectional area of the wire, when dealing with a macroscopic (ordinary, everyday life) wire. However, if the wire is, say, too short, e.g. comparable to the average distance a charge carrier travels before undergoing a collision, then it might begin to depend on the wire length, but for all practical intents and purposes a wire won't be that short.
The reason v does not depend on the wire cross sectional area is that the ratio I/A is constant (assuming the applied electric field within the wire is not changing), also called the current density, denoted by J=I/A. So, for example, if A doubles, I will also double (wire capacity doubles), keeping J constant.
Now according to Ohm's law $\mathbf J=\sigma\mathbf E$, as current density changes electric field should also change with cross sectional area. How is this possible if electric field is a property of point in space and we maintain a uniform field across the conductor.
We can't maintain uniform electric field across such a conductor, because its resistance is non-uniform.
How electric field changes with cross sectional area?
If you have cross section $A$ depending on distance $d$ from the origin as
$$A = A(d)$$
the resistance will depend on distance too:
$$\textrm dR(d) = \frac 1 \sigma \frac{\textrm d \ell}{A(d)}$$
where $\textrm d\ell$ is a linear element of the conductor and $\textrm dR(d)$ is its resistance.
Since
$$V = IR$$
on a linear element $\textrm d\ell$ there will be a drop of potential $\textrm d\varphi(d)$
$$\textrm d\varphi(d) = I \frac 1 \sigma \frac{\textrm d \ell}{A(d)}$$
Since
$$\int\limits_a^b E\,\textrm d\ell = V$$
we can write for the difference of potentials on an element $\textrm d\ell$
$$E\, \textrm d\ell = \textrm d\varphi(d)$$
Therefore
$$E = I \frac 1 {\sigma {A(d)}}$$
Hopefully this shows the intuition behind the electric field not being uniform.
Best Answer
Your interpretation makes intuitive sense, because you're assuming that $v$, the drift velocity of the charge carries, is the same at all points along the wire. That assumption isn't right.
Water flowing through pipes (or a river) is a common analogy: A wide spot in the conductor is like a wide tank that the pipe runs into one end of and out the other end of. The water will flow more slowly inside the tank than the pipe precisely such that the volume of water passing through each section per second is the same.
If the flux of water volume (or the the electrical current $I$) were higher in one section than the section just before it, where would it be getting that extra water (those extra charge-carriers) from each second?
If the flux of water volume (or the the electrical current $I$) were lower in one section than the section just before it, where would the extra water (charge-carriers) from the prior section be going?