This issue is a bit confusing because there are two types of angular momentum. There's spin, where a rigid body rotates about an axis through its center of mass, and there's orbital, where the center of mass of a rigid body rotates about an axis. For example, the Earth spins about its axis and rotates around the Sun. The total angular momentum can always be decomposed into a sum of these two terms.
You probably have never heard of this, because the definition of the torque is just $\mathbf{r} \times \mathbf{F}$, which doesn't have any reference to "spin or orbital". But when you write the equation $\tau = I \alpha$, you're implicitly choosing to talk about one or the other, or else $\tau$, $I$, and $\alpha$ are meaningless. (For example, the Earth takes 1 day to spin but 1 year to orbit. So you can't just say "the" angular velocity of the Earth.)
If you're talking about spin, $I$ means the moment of inertia about the center of mass, $\alpha$ means the angular acceleration about an axis through the center of mass, and $\tau$ means the torque about the center of mass. There's no meaningful way to change axes or origin because it's always the center of mass.
Now let's talk about the orbital part. The instantaneous angular velocity can always be defined, even if the object isn't moving in a circle about some point, using the equation
$$\mathbf{v} = \boldsymbol{\omega} \times \mathbf{r}$$
However, you can see this quantity is frame dependent: if we just move the origin, $\mathbf{r}$ will change but $\mathbf{v}$ won't, so $\boldsymbol{\omega}$ will change. As an example, if you see an airplane from the ground, its angular velocity appears very low, but if you're hovering next to it, it zips around really fast. So in this case, everything does change.
Okay, maybe neither of these examples were really what you wanted: the first was trivial, and the second didn't say much. We can get more insight by not doing the spin-orbital decomposition at all, which requires tossing out $\boldsymbol{\omega}$ and $\alpha$. The only rotational equation we have left is
$$\tau = \frac{dL}{dt}$$
which expanded out is
$$ \sum \mathbf{r} \times \mathbf{F} = \frac{d}{dt}\left( \sum \mathbf{r} \times \mathbf{p}\right)$$
Let's transform to another frame. To make it easy, let's just shift the origin by $\mathbf{r}_0$. Now if physics works, the resulting equation should be equivalent.
Let's confirm it. In the other frame, we have
$$ \sum (\mathbf{r} + \mathbf{r}_0) \times \mathbf{F} = \frac{d}{dt}\left(\sum (\mathbf{r} + \mathbf{r}_0) \times \mathbf{p}\right)$$
If we subtract out our previous equation, we're left with
$$ \sum \mathbf{r}_0 \times \mathbf{F} = \frac{d}{dt} \left( \sum \mathbf{r}_0 \times \mathbf{p} \right)$$
Since $\mathbf{r}_0$ is constant, the derivative only acts on $\mathbf{p}$, giving $\sum \mathbf{r}_0 \times \mathbf{F}$ on the right hand side. So the equation is true. Physics works!
Here's one way to think about why they transform seemingly oppositely. Force and velocity form a pair in the sense that when multiplied you get power:
$$W = \int F \cdot v\,dt.$$
The same is true for torque and angular velocity:
$$W = \int \tau \cdot \omega\,dt.$$
So if to go from force to torque we multiply by $r$, we need to compensate by dividing velocity by $r$.
Best Answer
Here is the proof that I think you're looking for. As Ali remarks in his answer, the results holds true for a rigid body undergoing rotation with constant angular velocity.
Let $\vec r_i$ denote the position of some particle in a rigid body. Suppose this rigid body is undergoing rotation with angular velocity $\vec \omega$, then $$ \dot {\vec r}_i = \vec \omega\times\vec r_i $$ See the appendix for a proof of this. By taking the derivative of both sides with respect to time and multiplying both sides by $m_i$, the mass of particle $i$, we obtain $$ \dot {\vec p}_i = \omega\times \vec p_i $$ Now we simply note that if $\vec F_i$ denotes the net force on particle $i$, then Newton's Second Law gives $\vec F_i = \dot{\vec p_i}$ so that \begin{align} \vec\tau_i &= \vec r_i\times \vec F_i \\ &= \vec r_i\times\dot{\vec p_i} \\ &= \vec r_i\times(\vec\omega\times\vec p_i) \\ &= -\vec p_i\times(\vec r_i\times \vec\omega) - \vec\omega\times(\vec p_i\times\vec r_i) \\ &= \vec p_i\times(\vec\omega\times\vec r_i) + \vec\omega\times(\vec r_i\times\vec p_i) \\ &= \vec p_i\times \dot{\vec r}_i + \vec \omega\times \vec L_i \\ &= \vec\omega\times \vec L_i \end{align} This is basically the identity you were looking for. In the fourth equality, I used the so-called Jacobi identity. Now, by taking the sum over $i$, the result can readily be seen to also hold for the net torque $\tau$ on the body and the total angular momentum $\vec L$ of the body; $$ \vec \tau = \vec\omega\times\vec L $$
Appendix. The motion of a rigid body undergoing rotation is generated by rotations. In other words, there is some time-dependent rotation $R(t)$ for which $$ \vec r(t) = R(t) \vec r(0) $$ It follows that $$ \dot{\vec r}(t) = \dot R(t) \vec r(0) = \dot R(t)R(t)^T\vec r(t) = \vec\omega(t)\times \vec r(t) $$ In the last step, I used the fact that $R(t)$ is an orthogonal matrix for each $t$ which implies that $\dot R R^T$ is antisymmetric. It follows that there exists some vector $\vec \omega$, which we call the angular velocity of the body, for which $\dot R R^T \vec A = \vec \omega\times\vec A$ for any $\vec A$.