To make it easier to analyze, let's decompose this 'reaction' force on the pivot in directions perpendicular $(x')$ and parallel $(y')$ to the bar (instead of the indicated $x$ and $y$ you indicated). Let's call these components $F_x'$ and $F_y'$.
Equations for translational motion:
$$F_y' - P_y' = 0$$
$$F_x' - P_x' = m*a$$
Now, let's calculate the torque about the center of mass, assuming the pendulum's length is L:
$$T(cm) = -F_x'*L/2 = I(cm)*b$$
Where $b$ is the angular acceleration of the pendulum about the center of mass. $I(cm)$ for a pendulum is equal to $mL²/12$. Hence:
$$F_x' = -mL*b/6$$
The final problem is determining the relationship between $b$ and $a$. That is easily done considering the center of mass rotates in a circular motion about the pivot, hence:
$$a = b*L/2$$
$L/2$ being the radius of the circumference described by the pendulum's center of mass. Making the substitution:
$$-mLb/6 - P_x' = mLb/2$$
Remember that $P_x' = mg*sin(\theta), \theta = angle$ between the pendulum and the vertical direction $y$.
- $mgsin(\theta) = -2mLb/3$
Which is the same differential equation you would obtain considering the torque about the pivot? Don't forget $b$ is the second time derivative of $\theta$.
I hope this clarifies your doubts a bit. As for the nature of these forces, they are of electromagnetic origin (just as any normal force) caused by the bond strength of whichever materials are in contact at the pivot. And yes, I am considering there is no attrition and hence energy dissipation.
There is a torque, but it points sideways, perpendicular to the orientation of the hammer at any time. Because of this, as the hammer rotates, the direction of the torque also rotates around with it. The torque only acts to rotate the system horizontally around in space, not to change the direction of its angular velocity.
Let's see this with a calculation. Suppose I model the hammer as a rod of length $L$ and mass $m_r$ with a point mass $m_p$ on the end.
The moment of inertia of the hammer at this moment can be computed by taking the moment of inertia of a similar configuration aligned along the $x$ axis and rotating it by an angle $-\theta$ in the $y$ direction:
$$\begin{align}I &= R_y^{-1}\begin{pmatrix}0 & 0 & 0 \\ 0 & \frac{mL^2}{3} + m_pL^2 & 0 \\ 0 & 0 & \frac{mL^2}{3} + m_pL^2\end{pmatrix}R_y \\
&= \begin{pmatrix}ML^2\sin^2\theta & 0 & -ML^2\sin\theta\cos\theta \\ 0 & ML^2 & 0 \\ -ML^2\sin\theta\cos\theta & 0 & ML^2\cos^2\theta\end{pmatrix}\end{align}$$
where $R_y$ is the rotation matrix around the $y$ axis and $M = \frac{m_r}{3} + m_p$. Computing the angular momentum using $\vec{L} = I\vec\omega$, where $\vec\omega = \omega\hat{z}$, I get
$$\vec{L} = ML^2\cos\theta(\hat{z}\cos\theta - \hat{x}\sin\theta)$$
The torque, on the other hand, is
$$\vec\tau = \vec{r}\times\vec{F} = (\hat{x}\cos\theta - \hat{z}\sin\theta)\times (-mg\hat{z}) = mg\hat{y}\cos\theta$$
So the torque actually pushes the angular momentum in a direction perpendicular to both its direction and the orientation of the hammer. This means there will be no change in the amount of angular momentum. It also means that the hammer rotates along with everything else, so that the angular momentum and the momenta of inertia preserve their relative orientation and thus there will be no change in its angular velocity.
Best Answer
There is a easy semi-geometrical way of finding the center of rotation due to a force.
Point R is the instance center of rotation.
See also https://physics.stackexchange.com/a/212939/392 for more details.
Example
A plank of length $a$ and and width $b$ is struck on one end with a force $F$. Place a coordinate system at the center of mass and measure the force moment arm as $c=\frac{a}{2}$. The mass moment of inertia of a plank is $$I_C = \frac{m}{12} \left( a^2+b^2\right)$$ and hence the radius of gyration about the center is $$\rho = \sqrt{ \frac{a^2+b^2}{12} }$$
With the method above the center of rotation is at a distance $$\boxed{\ell = \frac{\rho^2}{c} = \frac{ \frac{a^2+b^2}{12} } {\frac{a}{2} } = \frac{a^2+b^2}{6 a} }$$
Lets find the same answer using the equations of motion.
Same answer!! No need to do equations of motion to get the center of rotation. The point is purely a result of the inertial properties and geometry.