A pure torque does not have a point of application. It is shared among the entire rigid body. Only torque as as result of a force at a distance needs specification of the point of measurement.
For the equations of motion you need the net torque about the center of mass $\vec{T}_C$. If this torque is a result of an applied pure torque $\vec{\tau}$ and a force $\vec{F}$ located at a point A specified by $\vec{r}_{A/C}$ relative to the center of mass then $$\boxed{\vec{T}_C = \vec{\tau} + \vec{r}_{A/C} \times \vec{F}}$$
To see the equations of motion expressed on a point different from the center of mass read this answer about Derivation of Newton-Euler equations of motion not at the center of mass.
$$ \begin{aligned}
\sum \vec{F} &= m \vec{a}_A - m \vec{c}\times \vec{\alpha} + m \vec{\omega}\times\vec{\omega}\times\vec{c} \\
\sum \vec{T}_A &= I_C \vec{\alpha} + m \vec{c} \times \vec{a}_A - m \vec{c} \times \vec{c} \times \vec{\alpha} +\vec{\omega} \times I_C \vec{\omega} + m \vec{c} \times \left( \vec{\omega} \times \vec{\omega} \times \vec{c} \right)
\end{aligned} $$
In this answer $\vec{T}_A$ is the torque at a point not at the center of mass.
Suppose a ball moves horizontally to the right and hits a vertical stick at a position closer to its upper end. I understand intuitively, the stick will have linear and angular momentum, and that it will spin around its center of mass.
For simplicity, let's assume following the collision the ball ends up at rest, so that all of its momentum and angular momentum is transferred to the stick. While the specifics of how we would get this to happen depends on the mass and velocity of the ball, and where it hits the stick, but let's just assume it has been set up this way (or that there are other internal interactions that allow this to happen). If you want to consider cases where the ball does not end up at rest after the collision then you can do that, but I don't think it is essential for the question and answer at the moment.
Let's also assume that the collision is quick, so we don't need to get into the mess of thinking about the forces during the collision.
But why must the mass rotate around its center of mass? Why couldn't the stick simply move horizontally to the right (along with the ball) and not spin at all since the stick's center of mass isn't pinned to anything? It seems like there exists an imaginary pin at the center of mass causing it to rotate around that point regardless of where the ball hit.
We can pick our reference point to be the center of mass of the object. Then the ball has some non-zero angular momentum about the reference point, and since the ball comes to rest it must be that the stick picks up angular momentum about this point. If the stick didn't spin, then we would have no angular momentum about the center of mass. So we at least agree that there has to be some sort of rotation.
But why should the object spin about the center of mass? Well, if the object spun about some other point, then there would have to be some net force being applied to the object (with the center of mass of the object still being the reference point).$^*$ But after the collision, there are no forces acting on the stick, so we cannot have a torque acting on the stick. Continued rotation about the center of mass is the only way for the object's angular momentum to remain constant about the center of mass (our chosen reference point).$^{**}$
Can the point of reference be chosen at a point other than the center of mass (but still be on the stick), and calculate the torque about that point and still be able to calculate the stick's motion after collision? I've thought about picking the point the ball collides with the stick as the point of reference. Then by the definition of torque, the torque will be 0, which implies there will be no rotation at all? Why is this reasoning incorrect?
You are always free to pick any reference point you want to to calculate things like torque and angular momentum. You are right that if we choose the reference point to be the point of collision, then the ball exerts no torque on the stick. This just means that the angular momentum of the stick will be $0$ about this exact point in space. This does not mean the stick doesn't rotate, it just means that if you were to add up the angular momentum of each part of the stick as it moves (translational and rotational motion) you would find the total angular momentum to be $0$. Note that, once again, if there was no rotation of the stick at all, this condition would not be met.
If it is harder to wrap your head around this, then good. This is why we usually pick the reference point to be the point about which the object rotates, rather than some other point. The analysis becomes much easier to do.
$^*$Remember, $\mathbf a_\text{COM}=\mathbf F_\text{net}/M$, so if the object is rotating about a point that is not the center of mass, then $\mathbf a_\text{COM}\neq0$ and there must be a net force acting on the object.
$^{**}$Note that I have done a subtle change of reference frame here. Before the collision the reference point was the center of mass, which is at rest. After the collision, we are looking at the object in the reference frame of the object. The above analysis would still be true if we stayed in the original frame of reference, but I think it is easier to think of "before" and "after" the collision in these frames. The reason we can get away with this is because after the collision, the center of mass will move with a constant velocity.
Best Answer
Your intuition is correct.
If no external torque is applied, then the angular momentum of the thing doesn't change. That means, if one or both wheels start to rotate, the entire assembly will start to rotate in some way so that the total angular momentum does not change.
You can sometimes see this if you watch dirtbikes taking jumps. They can be in the air, not touching anything, and suddenly rotate forward so that the front wheel dips down. This happens if they hit one of the brakes in the air. One component (the wheel) stops rotating and loses angular momentum, then the entire assembly (bike and rider) starts rotating so that the net angular momentum of the entire assembly doesn't change.