The validity in $p$'s frame is yet another interesting coincidence. We just have to derive Newton's 2nd law for rotation relative to an inertial frame, but keeping track of the intermediate non-inertial frame. First, let $s$ be any particle in the object, and $g$ be any inertial frame. I'll use subscript pairs to indicate properties of one frame/point relative to another, and I'll use $\vec\tau$ for torque. Then we have
$\vec F_{s} = m_{s} \vec a_{sg} = m_{s}(\vec a_{sp} + \vec a_{pg}) \\
\vec r_{sp} \times \vec F_{s} = m_{s}(\vec r_{sp} \times \vec a_{sp} + \vec r_{sp} \times \vec a_{pg}) \\
\vec\tau_{sp} = m_{s}(\vec r_{sp} \times \frac{d}{dt}\vec v_{sp}) + m_{s}\vec r_{sp} \times \vec a_{pg} \\
\vec\tau_{sp} = m_{s}(\frac{d}{dt}(\vec r_{sp} \times \vec v_{sp}) - (\frac{d}{dt}\vec r_{sp}) \times \vec v_{sp}) + m_{s}\vec r_{sp} \times \vec a_{pg} \\
\vec\tau_{sp} = \frac{d}{dt}(\vec r_{sp} \times m_{s}\vec v_{sp}) + m_{s}\vec r_{sp} \times \vec a_{pg} \\
\vec\tau_{sp} = \frac{d}{dt}\vec L_{sp} + m_{s}\vec r_{sp} \times \vec a_{pg}$
Since $p$ is in general non-inertial, we now have the corrective term on the right. Now, sum over all particles:
$\Sigma_{s} \vec\tau_{sp} = \Sigma_{s}\frac{d}{dt}\vec L_{sp} + \Sigma_{s} (m_{s}\vec r_{sp} \times \vec a_{pg}) \\
\vec\tau_{net,p} = \frac{d}{dt} \vec L_{net,p} + (\Sigma_{s} m_{s}\vec r_{sp}) \times \vec a_{pg} \\
\vec\tau_{net,p} = \frac{d}{dt} \vec L_{net,p} + M_{tot}\vec r_{com,p} \times \vec a_{pg}$
Finally, rearranging so that the corrective term appears as the "fictitious torque" as mentioned in Claudio's answer:
$\frac{d}{dt} \vec L_{net,p} = \vec\tau_{net,p} + (\vec r_{p,com} \times M_{tot} \vec a_{pg})$
where $\vec r_{com,p}$ is the position of the center of mass relative to $p$. That's why it coincidentally disappears when you choose $p$ as the center of mass. But what if $p$ is the contact point on the object? Then, $\vec r_{com,p}$ is normal to the incline. The trickier part is $\vec a_{pg}$, but we do know that $\vec a_{pg} = \vec a_{p,com} + \vec a_{com,g}$. The motion relative to the center of mass is circular, so $\vec a_{p,com}$ has a tangential component up the incline, and a centripetal part normal to it. But the tangential part has magnitude $\frac{dv_{p,com}}{dt}$ where $v_{p,com}$ is the speed of $p$ relative to the COM. Meanwhile, $\vec a_{com,g}$ has magnitude $\frac{dv_{com,g}}{dt}$ down the slope. But since we are rolling without slipping, the speeds $v_{p,com}$ and $v_{com,g}$ are always equal (not just at this instant). Hence the tangential part of $\vec a_{p,com}$ cancels $\vec a_{com,g}$, leaving $\vec a_{pg}$ with only the centripetal part normal to the incline. This is parallel to $\vec r_{com,p}$, zeroing out their cross product. So the corrective term disappears in this frame, as you were hoping for.
As pointed out in a comment: all expressions of rotational dynamics can be constructed from the expressions of linear dynamics.
The only real difference is that linear dynamics is possible with one degree of spatial freedom, whereas dynamics requires a minimum of two spatial dimensions.
For example, there is the linear mechanics classroom demonstration device called 'air track'. The motion is effectively constrained to one spatial dimension.
For demonstration of circumnavigating motion, around some point of attraction of repulsion, an air table is used.
In the case of circular motion there are two ways of decomposing that motion that have practical use. One way is to decompose the motion along the axes of a rectangular grid. Then circular motion can be represented as a linear combination of two harmonic oscillations, 90 degrees out of phase.
Of course the decomposition used most often is according to polar coordinates: radial distance to the center of attraction/repulsion, and angle with respect to some reference line.
In both cases the motion is decomposed in two components. The component motions are at 90 degrees to each other.
Angular momentum
The concept of angular momentum had a precursor: Kepler's law of areas.
The very first proposition in Newton's Principia is a demonstration that Kepler's law of Areas follows logically from the laws of motion as laid down in the Principia.
I represented that derivation in an answer here on stackexchange to a question titled 'Intuition for angular momentum' The elements that go into that derivation are all from linear mechanics.
Point masses versus extended objects.
In mechanics when the motion of a solid object is modeled with an equation what is being tracked is the motion of the center of mass. Whatever the shape of some object A is, if it isn't touching any other objects (hence the motion is not affected) then that object A is effectively treated as a point mass. That is how it is possible to describe an extended object with a single value for its inertia; you treat it as if it is a point mass.
In angular mechanics there is the concept of Moment of inertia. If you have a wheel you can use the moment of inertia of that wheel in a calculation.
That raises the question: there is no way to treat that wheel as a point mass, it is inherently an extended object. How can you assign a single value to the moment of inertia?
The entities of angular mechanics are constructed by capitalizing on rotational symmetry.
A wheel is (for the purpose of the calculation) rotationally symmetric. For simplicity we count only the mass of the rim (treating the mass of the spokes as negligable). For the magnitude of the moment of inertial we treat all of the mass of the rim as if concentrated in a point at some distance 'r' to the center of rotation. This is valid because all that counts for the magnitude of the moment of inertia is the distance to the center of rotation.
Best Answer
Torque equals the rate of change of angular momentum. Differentiate the angular momentum to get the answer. And your method is not correct. Don't treat vectors like scalars. That is simply not allowed.
$$\vec{L} = I\vec{\omega} $$
Using your method, you would need to differentiate this way
$$\tau = \frac{d }{dt} (\vec{r} \times m \vec{v})$$