Since this is a homework problem, I will only provide a sketch of the solution. From the conservation laws, we have the three equations
$$\begin{align}
\tag{1} m_1v_{1i} - m_1v_{1f}\cos \theta &= m_2v_{2f} \cos \phi, \\
\tag{2} m_1v_{1f}\sin \theta &= m_2v_{2f}\sin \phi, \\
\tag{3} m_1v_{1i}^2 - m_1v_{1f}^2 &= m_2v_{2f}^2.
\end{align}$$
Summing the squares of (1) and (2) eliminates $\phi$. The RHS of the resultant equation contains $v_{2f}^2$ which can be eliminated using (3). Then, one would obtain a quadratic equation in terms of $\frac{v_{1f}}{v_{1i}}$, which can be solved to obtain the desired equation
$$\frac{v_{1f}}{v_{1i}} = \frac{m_1}{m_1+m_2}\left[\cos \theta \pm \sqrt{\cos^2 \theta - \frac{m_1^2-m_2^2}{m_1^2}}\right].$$
For the next equation, we rotate the axes to obtain the angle $\theta+\phi$ more easily. Here, the conservation laws are
$$\begin{align}
\tag{4} m_1v_{1i}\cos\phi - m_1v_{1f}\cos(\theta+\phi) &= m_2v_{2f}, \\
\tag{5} m_1v_{1i}\sin \phi &= m_1v_{1f}\sin(\theta+\phi), \\
\tag{6} m_1v_{1i}^2 - m_1v_{1f}^2 &= m_2v_{2f}^2.
\end{align}$$
First, we square (4) and use (6) to eliminate $v_{2f}$. Then, we use (5) to eliminate $v_{1i},v_{if}$ from the resultant equation, obtaining an equation in terms of $\phi$ and $\theta+\phi$. Using trigonometric identities, we get an equation in terms of $\tan\phi$ and $\tan(\theta+\phi)$ only. Then, this equation can be rewritten as a quadratic equation in $\frac{\tan\phi}{\tan(\theta+\phi)}$:
$$\left[1-\frac{\tan\phi}{\tan(\theta+\phi)} \right]^2=\frac{m_2}{m_1}\left[1-\frac{\tan^2\phi}{\tan^2(\theta+\phi)}\right],$$
which can be solved to obtain the desired equation
$$\frac{\tan(\theta +\phi)}{\tan(\phi)}=\frac{m_1+m_2}{m_1-m_2}.$$
I am going to ignore rotations in order to simplify the problem for your understanding. You have to enforce a series of inelastic relationship of the form
$$\vec{n}_{k}^\top (\vec{v}_i^+-\vec{v}_j^+) = 0 $$
where $\vec{n}_k$ is the normal direction of the $k$-th contact, and $i$, $j$ are the bodies this contact affects. The superscript $\phantom{c}^+$ denotes condition after the impact. You enforce this relationship with a series of $k$ impulses $J_k$ such that
$$ \vec{v}_i^+ = \vec{v}_i + \frac{\vec{n} J_k}{m_i} $$
$$ \vec{v}_j^+ = \vec{v}_j - \frac{\vec{n} J_k}{m_j} $$
Since it all has to happen at the same time it is best to form the problem with matrices.
Consider a Contact matrix $A$ where each column $k$ has +1 in the $i$-th row and -1 in the $j$-th row. For example $A = \begin{pmatrix}0&-1 \\ -1 & 0 \\ 0 & 0 \\ 1 & 1 \end{pmatrix}$ means there are two contacts, one between body 2 and body 4 and another between body 1 and 4. (actually each 0 and 1 are 2×2 for 2D or 3×3 for 3D zero and identity matrices).
The inelastic relationships are
$$N^\top A^\top v^+ =0$$ with the contact normal block diagonal matrix
$$ N = \begin{pmatrix} \vec{n}_1 & 0 & \cdots & 0 \\ 0 & \vec{n}_2 & & 0 \\ \vdots & & & \vdots \\ 0 & 0 & \cdots & \vec{n}_K \end{pmatrix} $$ and
$$ v = \begin{pmatrix} v_1 \\ v_2 \\ \vdots \\ v_N \end{pmatrix} $$
The momentum exchange is described by the relationship
$$ M v^+ = M v - A N J $$ where $M$ is the block diagonal mass matrix $M=\begin{pmatrix}m_1& & & \\ &m_2 & & \\& & \ddots & \\ & & & m_N\end{pmatrix}$ and $J$ the vector of impulses $J^\top=(J_1\,J_2\,\cdots J_K)$
To solve the problem we combine the momentum with the inelastic collisions to get
$$ v^+ = v - M^{-1} A N J $$
$$ N^\top A^\top \left(v - M^{-1} A N J \right) = 0$$
$$ \left(N^\top A^\top M^{-1} A N\right) J = N^\top A v $$
$$ \boxed{ J = \left(N^\top A^\top M^{-1} A N\right)^{-1} N^\top A^\top v }$$
Example
With $A$ as above (4 2D bodies, 2 contacts) and $\vec{v}_i = (\dot{x}_i,\dot{y}_i)^\top$, $\vec{n}_1=(1,0)^\top$, $\vec{n}_2 = (0,1)^\top$ then
$$ A = \left(\begin{array}{cc|cc} 0 & 0 & -1 & 0\\ 0 & 0 & 0 & -1\\ \hline -1 & 0 & 0 & 0\\ 0 & -1 & 0 & 0\\ \hline 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ \hline 1 & 0 & 1 & 0\\ 0 & 1 & 0 & 1 \end{array}\right) $$
$$ N = \left(\begin{array}{c|c} 1 & 0\\ 0 & 0\\ \hline 0 & 0\\ 0 & 1 \end{array}\right) $$
$$ M = \left(\begin{array}{cc|cc|cc|cc} m_{1} & 0 & 0 & 0 & 0 & 0 & 0 & 0\\ 0 & m_{1} & 0 & 0 & 0 & 0 & 0 & 0\\ \hline 0 & 0 & m_{2} & 0 & 0 & 0 & 0 & 0\\ 0 & 0 & 0 & m_{2} & 0 & 0 & 0 & 0\\ \hline 0 & 0 & 0 & 0 & m_{3} & 0 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & m_{3} & 0 & 0\\ \hline 0 & 0 & 0 & 0 & 0 & 0 & m_{4} & 0\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & m_{4} \end{array}\right) $$
$$ v = \begin{pmatrix} \dot{x}_1 \\ \dot{y}_1 \\ \hline \dot{x}_2 \\ \dot{y}_2 \\ \hline \dot{x}_3 \\ \dot{y}_3 \\ \hline \dot{x}_4 \\ \dot{y}_4 \end{pmatrix} $$
$$ N^\top A^\top M^{-1} A N = \left(\begin{array}{cc} \frac{1}{m_{1}} + \frac{1}{m_{4}} & 0\\ 0 & \frac{1}{m_{2}} + \frac{1}{m_{4}} \end{array}\right) $$
$$ N^\top A^\top v = \begin{pmatrix} \dot{x}_4-\dot{x}_2 \\ \dot{y}_4 - \dot{y}_2 \end{pmatrix} $$
$$ J = \begin{pmatrix} \frac{\dot{x}_4-\dot{x}_2}{\frac{1}{m_2}+\frac{1}{m_4}}
\\ \frac{\dot{y}_4-\dot{y}_1}{\frac{1}{m_1}+\frac{1}{m_4}} \end{pmatrix} $$
Then
$$v^+ = v - M^{-1} A N J = \begin{pmatrix}
\dot{x}_1 \\
\frac{m_1 \dot{y}_1 + m_4 \dot{y}_4}{m_1+m_4} \\
\frac{m_2 \dot{x}_2 + m_4 \dot{x}_4}{m_2+m_4} \\
\dot{y}_2 \\
\dot{x}_3 \\
\dot{y}_3 \\
\frac{m_2 \dot{x}_2 + m_4 \dot{x}_4}{m_2+m_4} \\
\frac{m_1 \dot{y}_1 + m_4 \dot{y}_4}{m_1+m_4} \end{pmatrix} $$
Appendix
To include rotations follow the guidelines here:
Best Answer
First of all, check your arithmetic on the X equation. You should get Vg(x) = +1.74 m/s.
Second, in the Y equation you should use the final orange speed 4.34 instead of 5.3 on the right hand side. You should get Vg(y) = -2.48 or so.