Find the intersections of the line containing the segment and the circle. This amounts to solving a quadratic equation. If there are no intersections (i.e. the solutions of the corresponding equation are non-real), then your segment does not intersect the circle. Now, if there are intersections, see whether they are inside the segment or not.
To implement this, let $P$ and $Q$ be the endpoints of your segment and $C$ and $r$ be the center and the radius of your circle. Then every point of the line through $P$ and $Q$ is given by the formula $$t P + (1-t) Q$$ for exactly one value of $t\in\mathbb R$, and the points in the segment are precisely those for which the corresponding $t$ is in the interval $[0,1]$.
Now the point corresponding to $t\in\mathbb R$ is on the circle if and only if $$\langle t P + (1-t) Q - C,t P + (1-t) Q - C\rangle = r^2,$$ where $\langle\mathord\cdot,\mathord\cdot\rangle$ is the usual inner (dot) product of vectors. If you solve this equation—this is easy, as it is a quadratic equation for $t$—, you find the $t$'s corresponding to the points of intersection of the circle and the line, and if at least one of those $t$'s belong to the interval $[0,1]$, then the segment intersects the circle.
Later: I tried to have Mathematica do the computation for me. Assuming $C=(0,0)$ and $r=1$, as we may up to translation and rescaling, and letting $P=(p1,p2)$, $Q=(q1,q2)$, the following computes the $t$'s:
P = {p1, p2};
Q = {q1, q2};
ts = t /. Solve[Norm[t P + (1 - t) Q]^2 == 1, t];
This ends up with the variable ts having the value
$$\left\{\frac{-p_1 q_1-p_2 q_2-\sqrt{p_1^2
\left(-q_2^2\right)-2 p_1 q_1+2 p_2 p_1 q_1
q_2-p_2^2 q_1^2-2 p_2
q_2+p_1^2+p_2^2+q_1^2+q_2^2}+q_1^2+q_2^2}{-2 p_1
q_1-2 p_2 q_2+p_1^2+p_2^2+q_1^2+q_2^2},\frac{-p_1
q_1-p_2 q_2+\sqrt{p_1^2 \left(-q_2^2\right)-2 p_1
q_1+2 p_2 p_1 q_1 q_2-p_2^2 q_1^2-2 p_2
q_2+p_1^2+p_2^2+q_1^2+q_2^2}+q_1^2+q_2^2}{-2 p_1
q_1-2 p_2 q_2+p_1^2+p_2^2+q_1^2+q_2^2}\right\}$$
(This was a huge formula, which will likely not fit on your browser window...)
If the expression inside the square roots is negative, there are no real $t$'s, so the line (hence, a fortiori the segment) and the circle are disjoint. If not, now we need to see if at least one of the roots in in $[0,1]$.
If I now tell Mma to tell me when then first of the roots is in $[0,1]$, by telling
her to compute
Reduce[0 <= ts[[1]] <= 1, Reals]
it works for a while, and comes up with a huge answer, presumably equivalent to the original
0 <= ts[[1]] <= 1
Can anyone make sense of the huge answer? (if one asks instead for the more meaningful
Reduce[0 <= ts[[1]] <= 1 || 0 <= ts[[2]] <= 1, t, Reals]
the same thing happens)
PS: Please notice that I have made absolutely no attempt at being fast or particularly smart with this idea: it is just the straightforward was to set up the problem and solve it.
Best Answer
Let's say that each horizontal cross-section of the cylinder (a circle) has a radius of $r$. That means that on any slice of the cylinder, we have a circle of the form
$$x_c^2+y_c^2=r^2 \text{(Eqn.1)}$$
Now let's write the equation of the line as if we were looking at the whole thing from above (i.e. ignoring it in the z-direction and writing $y_l$ as a function of $x_l$). It would have an apparent equation of the form $$y_l=mx_l+b \text{(Eqn 2)}$$ Now, if the line crosses the circle at at least one point, then $y_c=y_l$ and $x_c=x_l$. We substitute $y_l$ into $\text {Eqn 1}$ and $x_l$ into $\text {Eqn 1}$, making it
$$x_l^2+(mx_l+b)^2=r^2 \text{(Eqn.3)}$$ Now we can solve for $x_l$. We can plug that into $\text {Eqn 2}$ to find $y_l$, and because we know $y_c=y_l$ and $x_c=x_l$, we can solve for the other variables. If we know the location of $A$ and $B$, we can then find the position of that point in 3-dimensional space.