The question seems a bit odd because "time of maximum spring compression" is an odd concept. The spring compression is a function of time and the time of maximum spring compression is zero because it's an instant not a time interval. Maybe the question means the time interval from the time the car first touches the spring to the time of greatest compression.
Assuming this is the case, and bearing in mind that because this is a homework question we're only allowed to give hints, the trick to doing this question is to realise that the spring behaves as a simple harmonic oscillator i.e. the compression of the spring from the moment the car touches it will be:
$$d = A sin(\alpha t)$$
where $A$ and $\alpha$ are some constants that you need to calculate. The problem simplifies a lot if you think about the relation between the period of a harmonic oscillator and the amplitude of oscillation.
I am just repeating what everyone has said here, but hopefully this will make it clearer.
What you are doing is conflating $x$ as a coordinate and as a distance. When we use Newton's laws $F = m \ddot{x}$, $x$ is a coordinate.
Let's set up the scenario more carefully than you have done. Let's have a coordinate system where rightwards is positive. Say we have a spring of unstretched length $l$ (this is a distance, not a coordinate). Fix one end of the spring be at coordinate $0$, and let the free end be at coordinate $l$ initially.
Now we consider perturbations about this equilibrium configuration. Define $x$ to be the displacement of the free end from coordinate $l$, so that the coordinate of the free end is now given by $y = l + x$ (it is a function of time). Note that $x$ is not the extension amount. The extension amount is instead $|x|$.
Now we can do what you did, consider the two cases. When the spring is extended, that is, when $x > 0$, we have from Newton's 2nd law:
\begin{align}
m \ddot{y} = -k|x|.
\end{align}
Ok let's pause to see what I've written here. the LHS is the 2nd time derivative of the coordinate of the free end of the spring (with a mass attached to it). The right hand side is the sum of forces, which by Hooke's law has magnitude $k|x|$ and I have thrown in the minus sign because it points to the left.
But note that $\ddot{y} = \ddot{(l + x)} = \ddot{x}$ since $l$ is a constant. Also, since $x > 0$, $|x| = x$. So we get
\begin{align}
m\ddot{x} = -kx.
\end{align}
When the spring is compressed, $x < 0$. So
\begin{align}
m\ddot{y} = k|x|.
\end{align}
the LHS is exactly the same as before, and the RHS has now a positive sign because the force points to the right. But $x < 0$ means that $|x| = -x$, so we get
\begin{align}
m\ddot{x} = -kx,
\end{align}
exactly the same as before.
So there is no trouble.
Now you might protest and say, hey I want to define $x$ to be the distance of the free end from the equilibrium length, so $x \geq 0$, instead of as the displacement. Then what changes is this:
the coordinate is now a piecewise function. If the spring is stretched, $y = l + x$; if it is compressed, $y = l-x$.
For the stretched case, writing Newton's law gives
\begin{align}
m \ddot{y} = m\ddot{x} = -kx.
\end{align}
For the compressed case,
\begin{align}
m \ddot{y} = -m \ddot{x} = kx \implies m\ddot{x} = -kx.
\end{align}
Once again, we get back the same equations, and there is no problem.
Best Answer
You have found the velocity of body 2 after it is hit by the body 1 absolutely correctly. So let's consider the second part of the process, lasting from the moment of the hit until the moment of the maximum extension of a spring. Let's put that at initial time, velocity of the body 2 was: $$v_{0,a} = \frac{3}{4}v_0$$ Appropriately, velocity of the body 3 was zero: $v_{0,b} = 0$.
We need to determine somehow when the maximum spring extension would happen. The extension of the spring would be maximum, at the moment when the distance between the bodies is maximum. If the distance is maximum, then the separation(closing) speed is zero, it means that the angular velocity of the two bodies would be equal.
Let's denote the initial and the final angular velocities of the bodies as follows:
$$\omega_{0,a} = \frac{v_{0,a}}{2R} = \frac{3}{4\cdot 2R}v_0\tag{1}$$ $$\omega_{0,b} = 0$$ $$\omega_{1,a} = \omega_{1,b} = \omega_{1}$$
For the system of two bodies 2 and 4, the total angular momentum is the sum of angular momentum of each body: $$L=I_a\omega_a + I_b\omega_b$$ where $I_a$ and $I_b$ is the moment of inertia of the body 2 and 3 appropriately. For the initial time: $$L_0=I_a\omega_{0,a}$$ For the final time: $$L_1=(I_a + I_b)\omega_1$$ From the law of conservation of momentum, we know that $L_0 = L_1$, thus: $$\omega_1 = \frac{I_a}{I_a + I_b}\omega_{0,a}\tag{2}$$ Let's use the law of conservation of energy. First, let's find the energy of the system at the initial time: $$E_0=\frac{ I_a {\omega_{0,a}}^{2} } {2}$$ Energy of the system at final time: $$E_1= U + \frac{ I_a + I_b } {2}{\omega_1}^{2} $$ where $U$ - the potential energy of extended spring. Thus, from the law of conservation of energy: $$U = \frac{ I_a {\omega_{0,a}}^{2} } {2} - \frac{ I_a + I_b } {2}{\omega_1}^{2}$$ Let's express the $\omega_1$ in terms of $\omega_{0,a}$ by using the equation $(2)$: $$U = \left(\frac{ I_a } {2} - \frac{1}{2}\frac{ {I_a}^{2} } {I_a + I_b} \right) {\omega_{0,a}}^{2}\tag{3}$$ From the definition of the moment of inertia, we can get, that: $$I_a = 4mR^{2} $$ $$I_b = mR^{2} \tag{4}$$ Let's substitute the equations $(4)$ into equation $(3)$: $$U = \frac{2}{5}mR^{2} {\omega_{0,a}}^{2}\tag{5}$$ Let's express the $\omega_{0,a}$ in terms of $v_0$ by using the equation $(2)$: $$U = \frac{3^{2}}{4^{2}\cdot2\cdot5}m {v_0}^{2}\tag{6}$$ The potential energy of extended spring is: $$U=\frac{k{x}^{2}}{2}$$ where $x$ - is the extention of the spring and $k$ - is the stiffness of the spring. Thus, the extension would be: $$x=\sqrt{\frac{2U}{k}}\tag{7}$$ Eventually, substitute equation $(6)$ into the equation $(7)$:
$$x=\frac{3}{4}v_0\sqrt{\frac{m}{5k}}$$ So, the correct answer is B