Here are some questions to ask before building equations:
What is the shape of the bowl?
What is the mathematical description of the shape of the bowl?
Is the bowl massless?
How does the bowl swing? Does it swing from a string? Is that string massless?
Does the bowl rotate? (in addition to its swinging and having a ball roll on its surface)
From a related question, "Consider a solid ball of radius $r$ and mass $m$ rolling without slipping in a hemispherical bowl of radius $R$ (simple back and forth motion). "
"The only torque acting on the ball at any point in its motion is the friction force $f$. So we can write
$\tau = I\alpha = fr$
again using the rolling condition $a = r\alpha$ and the moment of inertia for a solid sphere,
$\frac{2}{5}ma = f$
The net force acting on the system is the tangential component of gravity and the force of friction, so
$F = ma = mgsin\theta - f$ "
Since your bowl is swinging, $\theta $ changes with time.
(Imagine the bowl is like a swinging pendulum bob)
Now lets discuss the details about the swinging bowl.
Consider a swinging bowl on a massless string of length $L'$ with period of oscillation $T'$ and maximum angular displacement $\theta_{max} '$
We need to form equations describing the change of the angle on inclination of the bowl with respect to the rolling bowl as the bowl swings.
Therefore, we need some initial condition. Let's say the bowl is at its maximum angular displacement to our 'left', and the hemispherical bowl always 'points' towards the 'axis' of it's swinging. Our 'total' inclination angle must not sum up to $\frac{\pi}{2} rad$, otherwise the ball would fall vertically instead of rolling without slipping.
First off, let's deal with $\theta$, the angle of inclination. Angle of inclination=angle of ball in bowl + angle of bowl in pendulum system.
Secondly, we need the equation describing the change of $\theta $ with time. Assume the bowl is massless but rigid and doesn't rotate (due to the other torque, exerted by the ball on the bowl). However, for a short while, let's imagine that the bowl does rotate. We would have some critical case (or range of cases) such that the rotation of the bowl corresponds to the swinging of the bowl in such a way that we can have large maximum angular displacements for the swinging of the bowl (perhaps even $2 \pi$, corresponding to full revolutions!). In other words, the rotation of the bowl could help increase the stability of the system.
If the bowl rotates, we need to even information about the bowl.
So in the oscillation of the bowl, we only consider the mass of the ball.
Rather, I think your first method is flawed. Because your $\alpha$ is always about the pivot you select. Since you select the contact point as pivot, then $\alpha$ should be about the contact point. So $\alpha$ is not $a/r$ of course.
Here is my approach for this problem.
First, I select the CM (center of mass) as the pivot. Let $f$ be the friction at the contact point. Every moment the total torque about the pivot is
$$\tau= fR= I \alpha$$
Here is another matter. How to determine $I$ ? My opinion is that since the shell is completely filled with frictionless fluid, then the fluid will not rotate with the shell at all since there is literally no force to push it to. So we ought to count only the shell's part. (I'm not definitely sure here, though) And thus $I=\frac{2}{3}MR^2$. Hence
$$f=\frac{I\alpha}{R}=\frac{2}{3}M R\alpha=\frac{2}{3} Ma$$
Then, consider the famous theorem of kinetic energy (which doesn't require conservation
!). Let $l$ be the distance the shell (or the CM, to be exact) has moved from the point where it is released, $v$ be the speed of the CM and $\omega$ be the angular speed. Note that the gravity and the friction are the only net forces acting on the shell together with the fluid contained, we have
$$(2Mg\sin\theta-f)l=\frac{1}{2}(2M)v^2+\frac{1}{2} I \omega^2 $$
Note that for a rotation without sliding, $v=R\omega$ always holds, thus the above equation simplifies to
$$(2Mg\sin\theta-f)l=\frac{4}{3}Mv^2$$
Differentiate both sides by $dt$
$$(2Mg\sin\theta -f)v=\frac{8}{3}Mv \frac{dv}{dt}=\frac{8}{3} Mva$$
Put $f=\frac{2}{3}Ma$ into it and note that $v$ cancels out, thus
$$a=\frac{3}{5}g\sin\theta$$
Best Answer
If $F_f=mg\sin\theta$ then there would be no net force down the slope and hence no linear acceleration of the ball down the slope.
Note that that the FBD is incorrectly drawn.
As the ball rolls down the slope without slipping the centre of mass of the ball undergoes a linear acceleration and there is also an angular acceleration of the ball.
As drawn there is no torque about the centre of mass of the ball and so there can be no angular acceleration of the ball.
The point of application of the frictional force $f$ must be moved as shown below.
In this case it is fairly obvious as to the direction of the frictional force but it is worth a little consideration as for some problems that direction is not quite as obvious eg a ball rolling up a slope.
If the ball slip down without rolling its acceleration would be greater than if the ball was rolling with no slipping.
In terms of energy the ball now converts its loss of gravitational potential energy into both linear and rotational kinetic energy so its final linear speed would be less when rolling.
That means when rolling the net force down the slope acting at the centre of mass must be smaller than when there is no rolling.
Thus the frictional force must act the slope.
The angular acceleration of the ball is in the clockwise direction hence the torque about the centre of mass must be clockwise again indicating that the frictional force is up the slope.
One can now set up two equations $F=ma$ and $\tau = I \alpha$ and with the no slipping condition $a=r \alpha$ solve a problem.
It is true that if the slope is too steep and/or the coefficient of static friction is too small the ball will roll and slip under the action of a kinetic frictional force ie the no slipping condition cannot be satisfied but usually this is not the case.
Perhaps you misheard your teacher?