Think about the definition of pressure:
$$P=\frac{F}{A}$$
Now, let's consider the definition of a force.
$$F=\frac{dp}{dt}=m\frac{dv}{dt}$$
Hence, for a given area and particle mass, the pressure is a function of the velocity:
$$P=\frac{m}{A}\frac{dv}{dt} $$
For starters, to quote Allan Pierce in Acoustics,
The often stated explanation, that oscillations in a sound wave are
too rapid to allow appreciable conduction of heat, is wrong.
That one surprised me when I learned it myself.
In fact, sound is not an adiabatic process for all frequencies. For any medium there is a thermal conduction frequency, $$ f_{\mathrm{TC}} =\frac{\rho c_{p} c^{2}}{2 \pi \kappa}. $$ Frequencies much lower than this value will be well-approximated as adiabatic. However, increasing the frequency through and above this point will transition the process from adiabatic to isothermal. For air, this frequency is $\sim 10^{9} \, \mathrm{Hz}$, well above the range of human hearing, so we almost always treat sound as adiabatic.
The physical reason this occurs is that heat transfer due to conduction is proportional to the temperature gradient. This is just a statement of Fourier's law for heat conduction. Consider what happens as the frequency of a harmonic wave decreases: The wavelength increases, and the slope of the oscillating waveform decreases as it is "stretched out." Assuming equal amplitudes, lower frequency waves will therefore set up smaller temperature gradients, which will conduct heat less effectively. If the heat conduction is negligible, then the entropy is conserved by the process.
So, in summary, the thermal gradients set up by sound waves for typical frequencies of interest are small enough to be neglected, hence sound is a (very nearly) adiabatic process. However, as Thomas pointed out below, in reality frequencies that cross into the potentially-isothermal regime are almost always affected by attenuation first, and the principal effects from conduction and viscosity are actually to damp out the sound wave.
In case you decide you do want to see some math, the energy equation is $$ \rho T \frac{d s}{d t} = \kappa \nabla^{2} T.$$ The previous arguments can be seen mathematically by linearizing about a quiescent base state and assuming harmonic wave solutions for $s$ and $T$. The equation can be rewritten as $$ - i \omega \rho_{0} T_{0} \hat{s} = - \kappa \frac{\omega^{2}}{c^{2}} \hat{T}, $$ $$ \hat{s} = - i \frac{\kappa \omega}{\rho_{0} T_{0} c^{2}} \hat{T}. $$ As the angular frequency $\omega = 2 \pi f \rightarrow 0,$ so must the amplitude of the entropy oscillation, $\hat{s}$. As with the quote at the beginning, much of my answer draws from Acoustics by Allan Pierce.
Best Answer
Remembering back to my undergrad days, and courtesy of a quick Google: if you define the acoustic potential $\phi$ by:
$$u = - \nabla \phi$$
then the pressure is:
$$p = \rho \frac{\partial\phi}{\partial t}$$
For a plane wave in one dimension the wave equation is:
$$\frac{\partial^2\phi}{\partial x^2} = \frac{1}{c^2} \frac{\partial^2\phi}{\partial t^2}$$
and the solution is:
$$\phi = f(ct \pm x)$$
for some function $f$. So $u$ and $p$ are in phase. However for a spherical wave the wave equation looks rather different:
$$\frac{\partial^2 (r\phi)}{\partial x^2} = \frac{1}{c^2} \frac{\partial^2 (r\phi)}{\partial t^2}$$
and you get:
$$\phi = \frac {f(ct \mp x)}{r}$$
where if I recall correctly the minus sign means a wave travelling outwards and the plus means a wave travelling inwards. So:
$$u = \frac {f(ct \mp x)}{r^2} \pm \frac {f'(ct \mp x)}{r}$$ $$p = \rho c\frac {f'(ct \mp x)}{r}$$
It's because $u$ and $p$ have a different $r$ dependance that you get the change in phase for distances below a wavelength or two.
Phew! Now you're going to ask me to put this into simple physical terms, but you'll have to let me go away and think about that. I guess it's basically because with a spherical wave the acoustic energy falls with distance in a sort of 1/$r^2$ way, but for a plane wave the acoustic energy is constant with distance.