Here are some questions to ask before building equations:
What is the shape of the bowl?
What is the mathematical description of the shape of the bowl?
Is the bowl massless?
How does the bowl swing? Does it swing from a string? Is that string massless?
Does the bowl rotate? (in addition to its swinging and having a ball roll on its surface)
From a related question, "Consider a solid ball of radius $r$ and mass $m$ rolling without slipping in a hemispherical bowl of radius $R$ (simple back and forth motion). "
"The only torque acting on the ball at any point in its motion is the friction force $f$. So we can write
$\tau = I\alpha = fr$
again using the rolling condition $a = r\alpha$ and the moment of inertia for a solid sphere,
$\frac{2}{5}ma = f$
The net force acting on the system is the tangential component of gravity and the force of friction, so
$F = ma = mgsin\theta - f$ "
Since your bowl is swinging, $\theta $ changes with time.
(Imagine the bowl is like a swinging pendulum bob)
Now lets discuss the details about the swinging bowl.
Consider a swinging bowl on a massless string of length $L'$ with period of oscillation $T'$ and maximum angular displacement $\theta_{max} '$
We need to form equations describing the change of the angle on inclination of the bowl with respect to the rolling bowl as the bowl swings.
Therefore, we need some initial condition. Let's say the bowl is at its maximum angular displacement to our 'left', and the hemispherical bowl always 'points' towards the 'axis' of it's swinging. Our 'total' inclination angle must not sum up to $\frac{\pi}{2} rad$, otherwise the ball would fall vertically instead of rolling without slipping.
First off, let's deal with $\theta$, the angle of inclination. Angle of inclination=angle of ball in bowl + angle of bowl in pendulum system.
Secondly, we need the equation describing the change of $\theta $ with time. Assume the bowl is massless but rigid and doesn't rotate (due to the other torque, exerted by the ball on the bowl). However, for a short while, let's imagine that the bowl does rotate. We would have some critical case (or range of cases) such that the rotation of the bowl corresponds to the swinging of the bowl in such a way that we can have large maximum angular displacements for the swinging of the bowl (perhaps even $2 \pi$, corresponding to full revolutions!). In other words, the rotation of the bowl could help increase the stability of the system.
If the bowl rotates, we need to even information about the bowl.
So in the oscillation of the bowl, we only consider the mass of the ball.
The following equation from @R. Romero's analysis is correct:
$$Mgh=\frac{1}{2}Mv^2+\frac{1}{2}I\left(\frac{v^2}{R^2}\right)\tag{1}$$But, the moment of inertia of a cylinder is given by: $$I=M\frac{R^2}{2}\tag{2}$$ So, combining Eqns. 1 and 2 gives:$$Mgh=\frac{1}{2}Mv^2+\frac{1}{4}Mv^2\tag{3}$$Calcelling M from both sides of the equation yields:$$gh=\frac{1}{2}v^2+\frac{1}{4}v^2\tag{4}$$So, solving for v, we have:$$v=\sqrt{\frac{4gh}{3}}$$Note that this is independent of the radius of the cylinder. So, both cylinders roll down the ramp in the same amount of time.
Best Answer
Friction force always opposes relative motion between two surfaces. In most cases (like yours), one of those surfaces is fixed. So, you should recognize that how does the other surface tend to move. For this purpose, you should consider that what force or torque want to move the body. Then, you can determine the correct direction of friction.
While ascending, the cylinder wants to roll because of its initial rotational inertia. So, friction force tries to oppose its rotation and act upwards the incline.
While descending, as it is fixed for a moment, so it doesn't have initial inertia. Force $mg\sin\theta$ wants to pull the cylinder downwards the incline. So, the cylinder tends to translate downwards and hence, friction force tries to opposes its translation and is upwards again.
(This answer can help you more!)