Have you considered finding the intersections using an implicit form for the circles, $$\frac{x^2}{r^2} + \frac{y^2}{r^2} + ax + by + c = 0?$$ This representation doesn't have any coefficients that diverge as the circle approaches a straight line. To find intersections, you'll have to solve a quadratic equation whose leading coefficient could be zero or arbitrarily close to it, but the alternative form of the quadratic formula should be able to deal with that robustly.
You'll then have to do some jiggery-pokery to figure out whether the intersection points lie within the arcs. If the arc's bending angle is smaller than $\pi$, a projection onto the line joining the endpoints will suffice.
(Disclaimer: While all of this feels like it should work, I haven't analyzed it in any detail. Also, there could still be a problem when the circle is close to a line and you want the longer arc. But I can't imagine that's a case that would turn up in any practical application.)
Update: For a concrete example, here is the equation for a circular arc passing through the three points $(0,0)$, $(0.5, h)$, and $(1,0)$: $$\kappa^2 x^2 + \kappa^2 y^2 - \kappa^2 x - 2\eta y = 0,$$ where $$\begin{align}\kappa &= \frac{8h}{4h^2 + 1}, \\ \eta &= \frac{8h(4h^2-1)}{(4h^2+1)^2}.\end{align}$$ As you can see, the coefficients remain bounded as $h \to 0$.
Update 2: Wait, that equation becomes trivial if $h = 0$, which is bad. We really want something like $x^2/r + y^2/r + ax + by + c,$ i.e. multiply the previous expression through by $r$. Then for the same example, our equation becomes $$\kappa x^2 + \kappa y^2 - \kappa x - 2\eta' y = 0,$$ where $\eta' = (4h^2-1)/(4h^2+1)$. Here are some explicit values.
$h = 1/2$: $$2 x^2 + 2 y^2 - 2 x = 0,$$ $h = 0.01$: $$0.07997 x^2 + 0.07997 y^2 - 0.07997 x + 1.998 y = 0,$$ $h = 0$: $$2 y = 0.$$
By the way, in this format, the linear terms will always be simply $-2(x_0/r)x$ and $-2(y_0/r)y$, where the center of the circle is at $(x_0,y_0)$. As the center goes to infinity but the endpoints remain fixed, these coefficients remain bounded and nonzero (i.e. not both zero).
[Collecting the discussion from the comments into an answer.]
You can certainly expand the expression that you’ve got and collect terms to get a quadratic equation in $\lambda$: Setting $\mathbf p = \mathbf o-\mathbf c$ and $\mathbf Q=\mathbf R^T\mathbf A\mathbf R$, you have $$(\lambda\mathbf l+\mathbf p)^T\mathbf Q(\lambda\mathbf l+\mathbf p) = (\mathbf l^T\mathbf Q\mathbf l)\lambda^2+2(\mathbf l^T\mathbf Q\mathbf p)\lambda +\mathbf p^T\mathbf Q\mathbf p=1,$$ which has real solutions when $(\mathbf l^T\mathbf Q\mathbf p)^2\ge(\mathbf l^T\mathbf Q\mathbf l)(\mathbf p^T\mathbf Q\mathbf p-1)$.
On the other hand, you’re essentially constructing the ellipsoid by scaling, rotating and translating the unit sphere. Since you’re not interested in the actual points of intersection, you could apply the inverse transformation to the line and then use the point-line distance formula to determine the line’s distance from the origin. If this is no greater than one, then there is at least one intersection.
If $\mathbf x_1$ and $\mathbf x_2$ are two points on a line, then the distance of the point $\mathbf x_0$ from this line is given by the formula (from Wolfram MathWorld) $${\lvert (\mathbf x_1-\mathbf x_0)\times(\mathbf x_2-\mathbf x_0)\rvert \over \lvert\mathbf x_2-\mathbf x_1\rvert}.$$ If $\mathbf x_0$ is the origin, this reduces to $${\lvert \mathbf x_2\times\mathbf x_1\rvert \over \lvert \mathbf x_2-\mathbf x_1\rvert}.\tag1$$ Now, since in your question you have $\mathbf R^T\mathbf A\mathbf R$ for the matrix of the ellipse, the matrix $\mathbf R$ already represents the inverse rotation. That is, it aligns the ellipsoid with the coordinate axes. (If instead you meant for it to represent a rotation that maps the standard basis onto the principal axes, then the ellipsoid’s matrix should be $\mathbf R\mathbf A\mathbf R^T$ instead and the inverse map below should use $\mathbf R^{-1}=\mathbf R^T$ instead of $\mathbf R$.) Setting $\mathbf S=\mathbf A^{1/2} = \operatorname{diag}(1/a,1/b,1/c)$, the required inverse map is therefore $\mathbf x\mapsto \mathbf S\mathbf R(\mathbf x-\mathbf c)$. A convenient choice for the two points on the line are the images of $\mathbf o$ and $\mathbf o+\mathbf l$. Plugging them into (1), we get $${\lvert \mathbf S\mathbf R(\mathbf o-\mathbf c)\times \mathbf S\mathbf R(\mathbf o+\mathbf l-\mathbf c) \rvert \over \lvert \mathbf S\mathbf R(\mathbf o-\mathbf c)-\mathbf S\mathbf R(\mathbf o+\mathbf l-\mathbf c) \rvert} = {\lvert \mathbf S\mathbf R(\mathbf o-\mathbf c)\times\mathbf S\mathbf R\,\mathbf l \rvert \over \lvert \mathbf S\mathbf R\,\mathbf l\rvert}.$$ Since we’re comparing a ratio of nonnegative numbers to $1$, we can square and rearrange to get $$\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\times\mathbf S\mathbf R\,\mathbf l \rvert^2 \le \lvert \mathbf S\mathbf R\,\mathbf l\rvert^2.$$ This looks like it could be more efficient to compute that the quadratic equation’s discriminant.
We can even go a bit farther. Using $\lvert\mathbf u\times\mathbf v\rvert^2=\lvert\mathbf u\rvert^2\lvert\mathbf v\rvert^2-(\mathbf u\cdot\mathbf v)^2$, $$\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\times\mathbf S\mathbf R\,\mathbf l\rvert^2=\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\rvert^2\lvert\mathbf S\mathbf R\,\mathbf l\rvert^2-\left((\mathbf S\mathbf R(\mathbf o-\mathbf c))\cdot\mathbf S\mathbf R\,\mathbf l\right)^2$$ so $$\left(\lvert\mathbf S\mathbf R(\mathbf o-\mathbf c)\rvert^2-1\right) \lvert\mathbf S\mathbf R\,\mathbf l\rvert^2 \le \left((\mathbf S\mathbf R(\mathbf o-\mathbf c))\cdot\mathbf S\mathbf R\,\mathbf l\right)^2,$$ but $\mathbf Q=(\mathbf S\mathbf R)^T(\mathbf S\mathbf R)$, so this is just the original inequality derived from the quadratic equation, refactored.
Note that if you already have unit direction vectors for the principal axes, there’s no reason for the purposes of this calculation to decompose the rotation into three primitive rotations as in your question: $\mathbf R$ is simply the matrix with these vectors as its rows. Note also that since $\mathbf S$ is diagonal, you can implement multiplication of a vector by $\mathbf S$ with three multiplications.
Best Answer
Just an idea - I don't know if it will fit your problem or not, cause actually you didn't describe it with sufficient level of details.
Let's think in terms of a cylindrical coordinate system - the equations below describe your cylindrical surface, ignoring that fact it has limited height (if I understood your question correctly).
$$x = R \cdot \sin(\phi), y = R \cdot \cos(\phi), z = z$$
Then you can substitute these $x$ and $y$ into the ellipsoid equation:
$$\frac {R^2} {a^2} \cdot \sin^2(\phi) + \frac {R^2}{b^2}\cdot \cos^2(\phi) + \frac{z^2}{c^2} = 1$$
You can express the $z$ as a function of $\phi$ - this will give you the intersection curve equation in cylindrical coordinates.
I hope it'll help.