I came across this question in a math textbook:
Prove that the vectors $a=3i+j-4k, b= 5i-3j-2k, c= 4i-j-3k$, are coplanar.
This was my attempt at a solution: If (a x b) x c = 0, then c is orthogonal to (a x b), so c is in the same perpendicular plane to (a x b) since (a x b) is perpendicular to both a and b, all three vectors are in the same plane perpendicular to (a x b). However, the problem is that the answer when using the formula doesn't equal to 0, which means that the vectors aren't coplanar. This means it's wrong because the logical conclusion of the proof is that the answer after using the formula comes out to zero.
Can someone show me the correct way to use this formula?
Best Answer
Remember that the definition of orthogonality (perpendicularity) is $\vec a$ is orthogonal to $\vec b$ if and only if $\vec a \cdot \vec b =0$.
$\vec a\times \vec b$ gives you a vector orthogonal to the plane of $\vec a$ and $\vec b$. Now you need to know if that vector is also orthogonal to $\vec c$. You don't want to use the cross product for that -- you want to use the dot product. So find $(\vec a\times \vec b)\cdot \vec c$. If that equals $0$, then $\vec c$ is in the same plane as $\vec a$ and $\vec b$.