In small cases, if you don't want to calculate the determinant, you can try to observe if one column can be written as a linear combination of the others, not just if it is a multiple of some other column. In your $3\times 3$ case, you have $C3=2C2-C1$, which shows the columns are a linearly dependent set, and so the matrix is not invertible. In the $2\times 2$ case, writing one column as a linear combination of the other is the same as seeing it is a multiple of the other, but this is not the strategy for more than two vectors.
Added: Geometrically, the other vector does not generally lie in the same direction as one of the other vectors, unless it is a multiple of just one vector, or if the other two vectors themselves are multiples of each other. If the two other vectors are a linearly independent pair, then in this case they span a two dimensional subspace isomorphic to the plane, which just looks like a plane passing through the origin at some angle. Since the third vector is in the span of the other two vectors, it will lie in that "plane" they span. As far as $n$-dimensional space, I have I hard time picturing it geometrically when $n\geq 4$. The general idea applies though that the vector will lie in the subspace spanned by the other vectors, but it doesn't necessarily have to be a multiple of any one vector.
You are confusing rays with vectors. Vectors represent length and direction. They are, thus, relative quantities. To say that two vectors aren't in the same plane is meaningless, because they have no origin. You can, however, say that two non-zero vectors always describe a plane.
Rays, on the other hand, have magnitude, direction, and origin, and certainly, the angle between them is well defined. So, your question seems to be more about whether the dot product can be used to find the angle between two rays, and the short answer is, yes.
While you are correct that two rays might not lie on intersecting lines, you can always find two parallel rays that do lie on intersecting lines. To see this, just draw a line segment from the origin of one ray, which we'll call ray A, to the origin of the other, which we'll call ray B. Then, draw another line segment of equal length, starting from the "tip" of ray A parallel to the first line segment. Finally, join the free ends of these two line segments, and you will get a ray parallel to ray A that shares its origin with ray B, as in the following crudely drawn picture:
Now, you have two co-planar rays, and you can use dot product confidently to your heart's content. Just make sure that you use the vectors of the rays, and not their origins.
Best Answer
What is the volume of the solid $[A,B,C,D]$? The answer is
$$ \left((B-A) \times (C-A)\right) \cdot (D-A) $$
Now $D$ will be in the plane if the volume is zero. So solve $$ \left((B-A) \times (C-A)\right) \cdot D = \left((B-A) \times (C-A)\right) \cdot A$$ You can simplify this to $$ \left(A\times B + B\times C+C\times A\right) \cdot D = (B\times C) \cdot A$$