\begin{cases}
t + xy &= a\,\quad(3)\\
t^2 + 2t xy+ x^2y^4 &= b\quad(4)\\
t^3 + 3t^2xy+ 3t\,x^2y^4+x^3y^9 &= c \quad(5)
\end{cases}
$xy=a-t\quad$ that we put into $(4)$ and $(5)$.
$\begin{cases}
t^2 + 2t (a-t)+ (a-t)^2y^2 &= b\quad(6)\\
\color{red}{t^3} + 3t^2(a-t)+ 3t\,(a-t)^2y^2+(a-t)^3y^6 &= c \quad(7)
\end{cases}$
from $(6)$ :
$$y^2=\frac{b-t^2 - 2t (a-t)}{(a-t)^2}=\frac{b+t^2 - 2at}{(a-t)^2} \quad(8)$$
We put $(8)$ into $(7)$ :
$$\color{red}{t^3} + 3t^2(a-t)+ 3t\,(a-t)^2\frac{b+t^2 - 2at}{3t\,(a-t)^2}+(a-t)^3\left(\frac{b+t^2 - 2at}{3t\,(a-t)^2} \right)^3=c$$
After simplification :
$$\alpha t^3+\beta t^2 +\gamma t+\delta = 0\quad
\begin{cases}
\alpha=c-3ab+2a^3 \\
\beta =3(b^2+a^2b-a^4-ac) \\
\gamma=3a(a^2b-2b^2+ac\\
\delta=b^3-a^3c
\end{cases}\tag 9$$
Then, solve the cubic equation for $t$.
Put $t$ into $(8)$. This gives $y$.
Then with $t$ and $y$ obtained, $\quad x=\frac{a-t}{y}$
Obviously the formulas for $t$, $y$ and $x$ will be very complicated.
Bringing in the language of inverse functions and so on isn't unreasonable, but in my opinion makes things more mysterious than they need to be. Instead I prefer a more set-theoretic, or perhaps "generalized geometric," interpretation.
The basic idea is that equations carve out geometric shapes in the relevant space, e.g. $\mathbb{R}^3$ - namely, their solution sets. Similarly, systems of equations then correspond to intersections: a system of equations describes the intersection of the shapes described by the individual equations in it. Algebraic forms correspond to geometric properties and vice versa, and this often lets us relate geometric and algebraic results: e.g. consider "three 'general' linear equations in three unknowns have a unique solution" versus "three planes in $\mathbb{R}^3$ in 'general position' have a single point in common."
Solving an equation, or system of equations then amounts to giving a "simpler" description of the corresponding set (and in particular, this simpler description should make it clear whether that set is nonempty). Note that this means that the solution process is "just" rephrasing. One slogan I like in this context is the following:
The equation becomes the answer.
The various tools we're "allowed" to use in solving a (system of) equation(s) correspond to theorems relating the solution sets given by certain related (systems of) equations, especially those which show that two equations have the same solution set:
The fact that e.g. adding something to both sides of an equation doesn't affect the solution set is a consequence of the basic rules of equality in first-order logic.
Other techniques are more context-specific: e.g. the fact that we can add "$a-a$" to one side of any equation relies on the particular axioms governing subtraction.
As a more complicated example, by the field axioms the solution set of $s=t$ is the union of the solution set of the equation ${s\over x}={t\over x}$ and the solution set of the system of equations $\{s=t, x=0\}$. Here we're not just asserting an equality between two solution sets, it's more complicated than that (and explains why division "feels different" as an equation-solving tool).
But "(systems of) equations are sets" is not the end of the story! There is in fact a second slogan which pushes us in a new direction:
Equations have lives of their own.
For example, we can consider "$4x^2-3y=17$" over $\mathbb{R}$, or over $\mathbb{C}$, or over $\mathbb{H}$, or over the integers modulo $42$, or etc.. Changing the structure changes the set associated to the equation, often leaving the realm of what we naively consider "geometry" altogether. The subject of algebraic geometry involves broadening our perspective on what constitutes "geometry" to include such things, though, and this broadening has turned out to be extremely useful.
In summary:
Equations, and systems of equations, describe ways of assigning sets, which we may try to think of as being shapes in some sense, to structures. Solving them (over a given structure) amounts to giving a nice description of the corresponding set.
And looking ahead to logic (everybody loves logic, right? :P), by generalizing this idea substantially at the cost of largely losing the geometric flavor we wind up with model theory - see e.g. here.
Best Answer
For your example, you have a "nearly" linear system, because only one equation is quadratic. In that case, you can express all solutions of the linear part of the system in the form $x_0+\alpha x_h$, substitute that expression into the quadratic equation and solve the resulting equation for $\alpha$.
If you apply this technique to your system, you get $v_e=v_a-\alpha$, $v_s=\frac{m_p}{m_s}\alpha$ and $\omega=\frac{lm_p}{2I}\alpha$. If you substitute this into the quadratic equation $$m_p (v_a^2-v_e^2) = m_s v_s^2 + I \omega^2$$ you get
$$m_p (2v_a-\alpha)\alpha = \frac{m_p^2}{m_s}\alpha^2 + \frac{l^2m_p^2}{4I}\alpha^2$$ One solution of this quadratic equation is obviously $\alpha=0$, but I guess you are more interested in the other solution. Assuming $m_p\alpha\neq0$, we can divide by $m_p\alpha$ to get $2v_a-\alpha = \frac{m_p}{m_s}\alpha + \frac{l^2m_p}{4I}\alpha$. It's easy to solve this equation for $\alpha$.