Using this approach you can derive a formula for solving quadratic equations. Let $a x^2 + b x - c$ be a polynomial with ($a \neq 0$), then:
$$a x^2 + b x = -c \iff a(x^2 + \frac{b}{a}x) = c$$
$$\iff a(x+\frac{b}{a}x +\frac{b}{2a}) = c + \frac{b}{2a}$$
$$\iff a(x+\frac{b}{2a})^2 = c + \frac{b}{2a}$$
$$\iff \pm\sqrt{a}(x+\frac{b}{2a}) = \sqrt{c+\frac{b}{2a}}$$
$$\iff x = \frac{\pm\sqrt{c+\frac{b}{2a}}}{\sqrt{a}} - \frac{b}{2a} $$
[Collecting the discussion from the comments into an answer.]
You can certainly expand the expression that you’ve got and collect terms to get a quadratic equation in $\lambda$: Setting $\mathbf p = \mathbf o-\mathbf c$ and $\mathbf Q=\mathbf R^T\mathbf A\mathbf R$, you have $$(\lambda\mathbf l+\mathbf p)^T\mathbf Q(\lambda\mathbf l+\mathbf p) = (\mathbf l^T\mathbf Q\mathbf l)\lambda^2+2(\mathbf l^T\mathbf Q\mathbf p)\lambda +\mathbf p^T\mathbf Q\mathbf p=1,$$ which has real solutions when $(\mathbf l^T\mathbf Q\mathbf p)^2\ge(\mathbf l^T\mathbf Q\mathbf l)(\mathbf p^T\mathbf Q\mathbf p-1)$.
On the other hand, you’re essentially constructing the ellipsoid by scaling, rotating and translating the unit sphere. Since you’re not interested in the actual points of intersection, you could apply the inverse transformation to the line and then use the point-line distance formula to determine the line’s distance from the origin. If this is no greater than one, then there is at least one intersection.
If $\mathbf x_1$ and $\mathbf x_2$ are two points on a line, then the distance of the point $\mathbf x_0$ from this line is given by the formula (from Wolfram MathWorld) $${\lvert (\mathbf x_1-\mathbf x_0)\times(\mathbf x_2-\mathbf x_0)\rvert \over \lvert\mathbf x_2-\mathbf x_1\rvert}.$$ If $\mathbf x_0$ is the origin, this reduces to $${\lvert \mathbf x_2\times\mathbf x_1\rvert \over \lvert \mathbf x_2-\mathbf x_1\rvert}.\tag1$$ Now, since in your question you have $\mathbf R^T\mathbf A\mathbf R$ for the matrix of the ellipse, the matrix $\mathbf R$ already represents the inverse rotation. That is, it aligns the ellipsoid with the coordinate axes. (If instead you meant for it to represent a rotation that maps the standard basis onto the principal axes, then the ellipsoid’s matrix should be $\mathbf R\mathbf A\mathbf R^T$ instead and the inverse map below should use $\mathbf R^{-1}=\mathbf R^T$ instead of $\mathbf R$.) Setting $\mathbf S=\mathbf A^{1/2} = \operatorname{diag}(1/a,1/b,1/c)$, the required inverse map is therefore $\mathbf x\mapsto \mathbf S\mathbf R(\mathbf x-\mathbf c)$. A convenient choice for the two points on the line are the images of $\mathbf o$ and $\mathbf o+\mathbf l$. Plugging them into (1), we get $${\lvert \mathbf S\mathbf R(\mathbf o-\mathbf c)\times \mathbf S\mathbf R(\mathbf o+\mathbf l-\mathbf c) \rvert \over \lvert \mathbf S\mathbf R(\mathbf o-\mathbf c)-\mathbf S\mathbf R(\mathbf o+\mathbf l-\mathbf c) \rvert} = {\lvert \mathbf S\mathbf R(\mathbf o-\mathbf c)\times\mathbf S\mathbf R\,\mathbf l \rvert \over \lvert \mathbf S\mathbf R\,\mathbf l\rvert}.$$ Since we’re comparing a ratio of nonnegative numbers to $1$, we can square and rearrange to get $$\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\times\mathbf S\mathbf R\,\mathbf l \rvert^2 \le \lvert \mathbf S\mathbf R\,\mathbf l\rvert^2.$$ This looks like it could be more efficient to compute that the quadratic equation’s discriminant.
We can even go a bit farther. Using $\lvert\mathbf u\times\mathbf v\rvert^2=\lvert\mathbf u\rvert^2\lvert\mathbf v\rvert^2-(\mathbf u\cdot\mathbf v)^2$, $$\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\times\mathbf S\mathbf R\,\mathbf l\rvert^2=\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\rvert^2\lvert\mathbf S\mathbf R\,\mathbf l\rvert^2-\left((\mathbf S\mathbf R(\mathbf o-\mathbf c))\cdot\mathbf S\mathbf R\,\mathbf l\right)^2$$ so $$\left(\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\rvert^2-1\right) \lvert\mathbf S\mathbf R\,\mathbf l\rvert^2 \le \left((\mathbf S\mathbf R(\mathbf o-\mathbf c))\cdot\mathbf S\mathbf R\,\mathbf l\right)^2,$$ but $\mathbf Q=(\mathbf S\mathbf R)^T(\mathbf S\mathbf R)$, so this is just the original inequality derived from the quadratic equation, refactored.
Note that if you already have unit direction vectors for the principal axes, there’s no reason for the purposes of this calculation to decompose the rotation into three primitive rotations as in your question: $\mathbf R$ is simply the matrix with these vectors as its rows. Note also that since $\mathbf S$ is diagonal, you can implement multiplication of a vector by $\mathbf S$ with three multiplications.
Best Answer
Let $u=A^{-1}a$, then: \begin{align}⟨a−Ax,B(a−Ax)⟩&=(-1)^2⟨A(x-A^{-1}a),BA(x-A^{-1}a)⟩ \\ &=(x-u)^TA^TBA(x-u) \\ &=(x-u)^T\frac 12(A^TBA+(A^TBA)^T)(x-u) \\ &=(x-u)^T\frac 12(A^T(B+B^T)A)(x-u) \\ &=(x-u)^TUDU^T(x-u) \\ &\le t \end{align} This is a quadratic form like an ellipsoid, translated by $u$, transformed by the orthogonal matrix $U$, and identified by the diagonal matrix $D$.