Conservation of angular momentum is closely related to the fact that there is no privileged spatial direction. As a consequence, once a direction is defined by an isolated system, it has no reason to change during the system evolution, because there is no distinguished absolute direction "calling it back" so to say. The system has a rotational inertia.
I will get back to this at the end of this discussion. But let's start with your intuition for linear momentum:
A single particle keeps drifting in a direction unless forced to
change.
This is because there is no privileged position in space. So, again, an isolated system that defines a position (by being there) has no reason to change it. Observed from another inertial frame, its center of mass moves uniformly in a straight line: this is how "to be not moving" appears in the most general sense. Else we would need to have a reference for absolute rest, which is precisely what we do not have because there is no absolute position.
So the system, having no reason to change the way it "not moves", has inertia: it takes a force to change the class of inertial frames it defines into another one. The force needed is the product of the eventual frame relative speed with respect to the initial one, with the system mass. The mass measures the inertia.
And a isolated group of particles too keep drifting as a whole
in an average direction unless forced to change. The law for an
isolated group of particles is exactly of the same form as that for a
single particle.
Yes, but what law are you talking about? Because conservation of momentum is not the conservation of speed: it is the conservation of the overall motion with inertia of the system (else it would require the same force to stop a fly than to stop a car with the same speed).
And this is why things seem different to you when it comes to angular momentum:
A particle cannot keep moving in a circle on it's own, whereas a
group of particles keep rotating as a whole
A group of particles does not keep rotating. See the case described in this question: we have two particles (astronauts there) linked by a taut rope and spinning around their centre of mass. When they let go of the rope, they each follow a straight trajectory: observed from an inertial (non-rotating) frame at rest with respect to their center of mass, their angular speed (with respect to their center of mass) progressively vanishes. Angular momentum is conserved because they move away from their center of mass, not because they keep moving in circle.
The law for a single particle leads to a law that is of a different form for the group.
The single particle is a solid object: its whole mass distribution is maintained by cohesive forces. If you abstractly divide it into different parts, each one of these parts would follow a straight line and move away from the center of rotation, if it was not bound to the other parts. Exactly as the astronauts do: they keep rotating as long as they are bound by the rope.
This has all been well explained in @stafusa's answer.
So the law of angular momentum conservation (that indeed is the same for a single particle and a group) is not the law you are thinking of: it is not a law about keeping moving in circle. It is not about conservation of the angular speed.
What is it then? It is a law about conservation of the overall direction with rotational inertia. Very much like a uniform linear motion is the most general way of "not moving", rotating in such a manner that the total angular momentum does not change is the most general way of "not rotating".
What is not analogous to the conservation of linear momentum is the (false) idea that we can characterize what is being conserved by a class of reference frames, because rotating frames are not inertial. So if we kept rotating along a many-body system, (in the same way as we previously followed a group of particles in inertial motion), we would not so easily observe that the system keeps "not rotating" (as opposed to the previous system center of mass "not moving"). We would have to take into account inertial forces (such as the Coriolis force) that seem to correspond to a change of angular speed, while they actually help preserve both rotation axis and rotational inertia.
Best Answer
Imagine you had a particle traveling at constant $\mathbf{v_0} = \dot{z_0}\mathbf{\hat{k}}$ upwards. At time $t = 0$, your particle is at $\mathbf{r} = z_0 \mathbf{\hat{k}}$ and you apply an horizontal force $\mathbf{F} = -F\mathbf{\hat{i}}$, constant $F$, for time $\Delta t$.
The particle was subect to a force, so its velocity was at every instant of application of the force: $$ \mathbf{v} = \mathbf{v}_0 + \Delta \mathbf{v} = \mathbf{v}_0 + \int_0^t\frac{d \mathbf{v}}{dt} dt = \mathbf{v}_0 + \int_0^t \mathbf{a} dt = \mathbf{v}_0 + \int_0^t \frac{\mathbf{F}}{m} dt $$ but the last integral really is just Newton's second law and: $$ \int_0^t \frac{\mathbf{F}}{m} dt = -\frac{F}{m} t \mathbf{\hat{i}} $$ and the final speed: $$ \mathbf{v}(t) = \dot{z_0}\mathbf{\hat{k}} -\frac{F}{m} t \mathbf{\hat{i}} \Longrightarrow \mathbf{v}(\Delta t) = \dot{z_0}\mathbf{\hat{k}} -\frac{F}{m} \Delta t \mathbf{\hat{i}} = \mathbf{v}'. $$ Its position is also given by integrating the last equation: $$ \mathbf{r}(t) = \mathbf{r_0} + \Delta \mathbf{ r} = (z_0 + \dot{z}_0 t)\mathbf{\hat{k}} - \frac{1}{2}\frac{F}{m}t^2 \mathbf{\hat{i}} \Longrightarrow \mathbf{r}(\Delta t) = (z_0 + \dot{z}_0 \Delta t)\mathbf{\hat{k}} - \frac{1}{2}\frac{F}{m}(\Delta t)^2 \mathbf{\hat{i}} = \mathbf{r'} $$ After the torque/force ends, it stops accelerating and therefore just travels rectilinearly through space with $\mathbf{v}'$. In the same coordinate system, you can check that angular momentum is, after calculating speed and position (reset the clock, so we can start at $t = 0$ again, that's licit, we "stored" everything that happened before on the $\Delta t$s, $\mathbf{r}'$ and $\mathbf{v}'$): $$ \mathbf{r}(t) = \mathbf{r}' + \mathbf{v}' t $$ $$ \mathbf{v}(t) = \mathbf{v}' $$ $$ \mathbf{L}(t) = \mathbf{r}(t) \times \mathbf{p}(t) = m \mathbf{r}(t) \times \mathbf{v}(t) = m(\mathbf{r}' + \mathbf{v}' t) \times \mathbf{v}' = m\mathbf{r}' \times \mathbf{v}' $$ notice that the primed vectors are constant, so this new angular momentum also is. So you really get rectilinear motion and constant angular momentum in the end.