There is a torque, but it points sideways, perpendicular to the orientation of the hammer at any time. Because of this, as the hammer rotates, the direction of the torque also rotates around with it. The torque only acts to rotate the system horizontally around in space, not to change the direction of its angular velocity.
Let's see this with a calculation. Suppose I model the hammer as a rod of length $L$ and mass $m_r$ with a point mass $m_p$ on the end.
![enter image description here](https://i.stack.imgur.com/NGid0.png)
The moment of inertia of the hammer at this moment can be computed by taking the moment of inertia of a similar configuration aligned along the $x$ axis and rotating it by an angle $-\theta$ in the $y$ direction:
$$\begin{align}I &= R_y^{-1}\begin{pmatrix}0 & 0 & 0 \\ 0 & \frac{mL^2}{3} + m_pL^2 & 0 \\ 0 & 0 & \frac{mL^2}{3} + m_pL^2\end{pmatrix}R_y \\
&= \begin{pmatrix}ML^2\sin^2\theta & 0 & -ML^2\sin\theta\cos\theta \\ 0 & ML^2 & 0 \\ -ML^2\sin\theta\cos\theta & 0 & ML^2\cos^2\theta\end{pmatrix}\end{align}$$
where $R_y$ is the rotation matrix around the $y$ axis and $M = \frac{m_r}{3} + m_p$. Computing the angular momentum using $\vec{L} = I\vec\omega$, where $\vec\omega = \omega\hat{z}$, I get
$$\vec{L} = ML^2\cos\theta(\hat{z}\cos\theta - \hat{x}\sin\theta)$$
The torque, on the other hand, is
$$\vec\tau = \vec{r}\times\vec{F} = (\hat{x}\cos\theta - \hat{z}\sin\theta)\times (-mg\hat{z}) = mg\hat{y}\cos\theta$$
So the torque actually pushes the angular momentum in a direction perpendicular to both its direction and the orientation of the hammer. This means there will be no change in the amount of angular momentum. It also means that the hammer rotates along with everything else, so that the angular momentum and the momenta of inertia preserve their relative orientation and thus there will be no change in its angular velocity.
Relevant equation for calculating the time rate of change of a vector between an inertial and non-inertial rotating frames of reference:
$$ \left(\frac{d\boldsymbol r}{dt}\right)_s = \left(\frac{d\boldsymbol r}{dt}\right)_r +\boldsymbol \omega\times\boldsymbol r$$
where $s$ denotes the space fixed frame and $r$ denotes the rotating frame.
That is the relevant equation for the instantaneous co-moving inertial frame. What about a non-comoving space frame? Generalizing this to a space frame in which the origin of the rotating frame is moving results in
$$ \left(\frac{d\boldsymbol r}{dt}\right)_s =
\left(\frac{d\boldsymbol r_0}{dt}\right)_s
+ \left(\frac{d(\boldsymbol r - \boldsymbol r_0)}{dt}\right)_r
+\boldsymbol \omega\times(\boldsymbol r - \boldsymbol r_0)$$
where $\boldsymbol r_0$ is the displacement vector from the origin of the space frame to the origin of the rotating frame.
Suppose you use some other point $\boldsymbol r_1$ that is fixed from the perspective of the rotation frame (i.e., $\left(\frac{d(\boldsymbol r_1-\boldsymbol r_0)}{dt}\right)_r \equiv 0$. Go through the math (an exercise I'll leave up to you) and you'll find that
$$ \left(\frac{d\boldsymbol r}{dt}\right)_s =
\left(\frac{d\boldsymbol r_1}{dt}\right)_s
+ \left(\frac{d(\boldsymbol r - \boldsymbol r_1)}{dt}\right)_r
+\boldsymbol \omega\times(\boldsymbol r - \boldsymbol r_1)$$
In other words, the angular velocity $\boldsymbol \omega$ is independent of the choice of origin.
Best Answer
Imagine that the rigid body is fixed in the $Oxy$ co-ordinate plane. The rigid body need not overlap the origin $O$. If the co-ordinate plane is rotated (eg about the origin $O$) then every line in the plane rotates through the same angle, regardless of its position. Each of the two points at the ends of any line rotates through the same angle about the other. This is true whether the two points both lie inside the rigid body, or one inside and one outside, or both outside.
Angular velocity is the rate of rotation, so what is true about angles applies also for angular velocity.
Consequently, when an axis is chosen outside of a rigid body, the angular velocity is only the same as that measured within the rigid body if the axis rotates as though it were part of the rigid body. For example, the whole co-ordinate plane could be regarded as a rigid body which has finite density inside the region occupied by the object of interest and zero density outside of the object in the rest of the plane.
The constant angular velocity $\omega$ between two points within the rigid body is intrinsic to the rigid body. It does not hold between two points one inside and one outside of the rigid body.
If the external axis (origin $O$) rotates relative to the rigid body this is equivalent to the rigid body rotating relative to the co-ordinate system $Oxy$. In this case, in general the angular velocity $\Omega_P$ of a point $P$ within the rigid body relative to $O$ is not only different from the intrinsic angular velocity $\omega$ within the rigid body, it is also not constant.
Suppose some point $C$ within the rigid body is at rest (perhaps instantaneously) in the $Oxy$ frame. Any point $P$ within the rigid body rotates about $C$ with constant angular velocity $\omega$. The angular velocity $\Omega_P$ of $P$ about $O$ varies as $P$ rotates about $C$. At $P_1, P_3$ the velocity of $P$ is aligned with the vector $OP$, so $\Omega_P$ is zero. At $P_2, P_4$ the point $P$ is aligned with vector $OC$ and $\Omega_P$ reaches a local maximum.
If the centre of rotation $C$ is rotating about $O$ then $\Omega_P$ is even more difficult to calculate except in special cases. One special case is when $C$ rotates about $O$ with the same angular velocity $\omega$ with which $P$ rotates about $C$. Then the angular velocity $\Omega_P=\omega$.