What does it actually mean for two bodies to "orbit their center of mass". Does that mean that the two bodies move in ellipses, and the center of mass is a foci of each ellipse?
It is also called a barycenter. Any two (or more) objects in orbit around each other all orbit the barycenter. When working with 2 objects, the center of mass is the barycenter. I think you are confusing the center of mass for each object with the center of mass for the system.
Suppose we are given two masses with non-collinear initial velocity vectors. Assuming the only relevant force is the gravitational attraction between the two masses, does it follow that the bodies orbit their center of mass? In other words, under what conditions do two bodies orbit their center of mass?
The 2 object must always orbit their center of mass of the system.
As for solving the question, it appears there is not enough information provided to determine the period.
Added (corrected):
The period can be found from:
$T=2\pi \sqrt {\frac{r^3}{G(M_1+M_2)}}$
The masses are M and 3M and the radii of the orbits is $\frac{1}{4}d + \frac{3}{4}d$.
Substituting the values gives:
$T=2\pi \sqrt {\frac{d^3}{4GM}}$
$T=\pi \sqrt {\frac{d^3}{GM}}$
Which is A.
The problem is equivalent to 4 spheres colliding simultaneously, where top sphere center is at $60^o$ relative to the $x'x$ axes (same goes for bottom sphere):
We'll name them: sphere A (dark blue), and spheres 1, 2, and 3.
During the collision the spheres will behave like springs with an infinite hook constant. The forces on the spheres will be infinite and the duration of interaction will be 0.
The red area is where the spheres are compressed during the collision. If sphere 2 is compressed by $l$, and spheres 1 and 3 by $d$, then $\frac{d}{l}=sin30^o$ (trigonometry involved but i couldn't draw it here) and finally $l=2d$.
The energy stored in each spring is given by the formula $U=\frac{1}{2}kx^2$, and it will be converted to kinetic energy. Therefor, $K_1=K_3=\frac{1}{4}K_2$ and $v_1=v_3=\frac{1}{2}v_2$.
Before the collision sphere A has velocity $v_A$ and after the collision, $v_B$.
Conservation of energy:
$K_A = \frac{1}{2}mv_A^2 = \frac{1}{2}m(v_B^2+v_1^2+v_2^2+v_3^2)$, that is $v_A^2=v_B^2+v_1^2+v_2^2+v_3^2=v_B^2+\frac{3}{2}v_2^2$ (1)
Conservation of momentum:
$mv_A=mv_B+mv_1+mv_2+mv_3$, that is, $v_A=v_B+v_{1x}+v_2+v_{3x}$
Momentum on y axes is 0, since $v_{1y}=v_{3y}$. Also, $v_1$ forms a $60^o$ angle, so $v_{1x}=\frac{1}{2}v_1=\frac{1}{4}v_2$ and
$v_B=v_A-\frac{3}{2}v_2$ (2)
Finally:
Using (1) and (2) we get a single solution $v_2=0.8v_A$. The other velocities can be easily calculated, since we have found $v_2$.
Best Answer
I don't exactly know what you mean by static forces. But I am going to take a wild guess here and assume that by that you mean forces involved in problems where bodies don't move. I think you assumed that Newton's second law quantifies a force. This is actually wrong. First of all realize that a force is an interaction and it still acts whether the body on which it acts moves or not. Newton's second law quantifies the total effect of all such forces on a body of mass $m$ and not the force itself. For example the Newton's law of Gravitation tells you that the force between two masses is:
$$\vec{F} = G\frac{Mm}{r^2}$$
Now this is practically useless unless you specifies what a force does on a body. That's where Newton's second law comes in. So along with Newton's second law, you have a complete theory of (classical) gravitation.
Also the $\vec{F}$ in Newton's second law is the total force acting on a body having momentum $\vec{p}$. So when bodies don't move the net forces on them is zero. But that does not mean that you can not have forces acting on it.