Since this is a homework problem, I will only provide a sketch of the solution. From the conservation laws, we have the three equations
$$\begin{align}
\tag{1} m_1v_{1i} - m_1v_{1f}\cos \theta &= m_2v_{2f} \cos \phi, \\
\tag{2} m_1v_{1f}\sin \theta &= m_2v_{2f}\sin \phi, \\
\tag{3} m_1v_{1i}^2 - m_1v_{1f}^2 &= m_2v_{2f}^2.
\end{align}$$
Summing the squares of (1) and (2) eliminates $\phi$. The RHS of the resultant equation contains $v_{2f}^2$ which can be eliminated using (3). Then, one would obtain a quadratic equation in terms of $\frac{v_{1f}}{v_{1i}}$, which can be solved to obtain the desired equation
$$\frac{v_{1f}}{v_{1i}} = \frac{m_1}{m_1+m_2}\left[\cos \theta \pm \sqrt{\cos^2 \theta - \frac{m_1^2-m_2^2}{m_1^2}}\right].$$
For the next equation, we rotate the axes to obtain the angle $\theta+\phi$ more easily. Here, the conservation laws are
$$\begin{align}
\tag{4} m_1v_{1i}\cos\phi - m_1v_{1f}\cos(\theta+\phi) &= m_2v_{2f}, \\
\tag{5} m_1v_{1i}\sin \phi &= m_1v_{1f}\sin(\theta+\phi), \\
\tag{6} m_1v_{1i}^2 - m_1v_{1f}^2 &= m_2v_{2f}^2.
\end{align}$$
First, we square (4) and use (6) to eliminate $v_{2f}$. Then, we use (5) to eliminate $v_{1i},v_{if}$ from the resultant equation, obtaining an equation in terms of $\phi$ and $\theta+\phi$. Using trigonometric identities, we get an equation in terms of $\tan\phi$ and $\tan(\theta+\phi)$ only. Then, this equation can be rewritten as a quadratic equation in $\frac{\tan\phi}{\tan(\theta+\phi)}$:
$$\left[1-\frac{\tan\phi}{\tan(\theta+\phi)} \right]^2=\frac{m_2}{m_1}\left[1-\frac{\tan^2\phi}{\tan^2(\theta+\phi)}\right],$$
which can be solved to obtain the desired equation
$$\frac{\tan(\theta +\phi)}{\tan(\phi)}=\frac{m_1+m_2}{m_1-m_2}.$$
In general, there is no solution to the outgoing velocities of the particles. You need 6 components (three components of velocities for each particles) but you have only 4 equations (three components from the conservation of momentum, one from the conservation of energy). There are 2 equations missing.
To resolve this problem, you have to define what happens microscopically during the collision. For example, you can use a known potential for the interaction between the two particles and derive the trajectories.
Best Answer
For energy conservation, the directions of the vectors are not important, as energy is a scalar quantity. For the kinetic energy you can simply plug in everything you have in the text into the equation you stated - as long as the collision is elastic. The directions only matter for the conservation of momenta, this is $$m_1v_{1i}+m_2v_{1i}=m_1v_{1f}+m_2v_{2f}\,,$$ where you need to take care of the directions of the vectors, i.e., the direction of the momenta.
Sometimes it is useful to combine conservation of energy and conservation of momenta to solve for unknown quantities. See for example https://en.wikipedia.org/wiki/Elastic_collision
But be careful - kinetic energy is in general not conserved for inelastic collisions.