If I have two 3d vectors then I can use the dot product to find the angle between them. Since cosine inverse returns a value between $0^\circ$ and $180^\circ$, there are two vectors that could have had the same dot product value. If I want to rotate one vector to match the other I need to know whether to rotate $-\theta$ or $\theta$. How do I find out which direction to rotate? This image explains what I mean:
Best Answer
3D:
Let's say you have two vectors, $\vec{a}$ and $\vec{b}$. They are perpendicular to $\vec{n}$, $$\vec{n} = \vec{a} \times \vec{b} \tag{1}\label{NA1}$$ If we measure the rotation around the vector $\vec{n}$, the angle between the two vectors is $\varphi$, $$\cos\varphi = \frac{\vec{a} \cdot \vec{b}}{\left\lVert\vec{a}\right\rVert \, \left\lVert\vec{b}\right\rVert} \tag{2}\label{NA2}$$
In general, we can also utilize $$\sin\varphi = \pm \frac{\left\lVert \vec{a} \times \vec{b} \right\rVert}{\left\lVert\vec{a}\right\rVert\,\left\lVert\vec{b}\right\rVert} \tag{3}\label{NA3}$$ where the sign depends on the orientation (clockwise or counterclockwise) we measure the rotation (around the plane normal $\vec{n}$) from $\vec{a}$ to $\vec{b}$.
If you have a specific unit direction vector $\hat{d}$, either parallel or opposite to $\vec{n}$, $$\hat{d} = \pm \frac{\vec{n}}{\left\lVert\vec{n}\right\rVert} = \pm \frac{\vec{a}\times\vec{b}}{\left\lVert\vec{a}\times\vec{b}\right\rVert}$$ then you can calculate the angle counterclockwise around $\hat{d}$ using $\eqref{NA2}$ and $$\sin\varphi = \frac{\hat{d}\cdot(\vec{a}\times\vec{b})}{\left\lVert\vec{a}\right\rVert\,\left\lVert\vec{b}\right\rVert}$$
2D:
In this case, there is only one "side"; the 2D coordinate plane. Here, we can use the 2D analog of the vector cross product: $$\begin{array}{l} \vec{a} = ( x_a , y_a ) \\ \vec{b} = ( x_b , y_b ) \\ \vec{a} \cdot \vec{b} = x_a x_b + y_a y_b \\ \vec{a} \times \vec{b} = x_a y_b - x_b y_a \end{array}$$ and $$\begin{cases} \cos\varphi = \frac{\vec{a} \cdot \vec{b}}{\left\lVert\vec{a}\right\rVert \, \left\lVert\vec{b} \right\rVert} = \frac{ x_a x_b + y_a y_b}{\sqrt{(x_a^2 + y_a^2)(x_b^2 + y_b^2)}} \\ \sin\varphi = \frac{\vec{a} \times \vec{b}}{\left\lVert\vec{a}\right\rVert \, \left\lVert\vec{b} \right\rVert} = \frac{ x_a y_b - x_b y_a}{\sqrt{(x_a^2 + y_a^2)(x_b^2 + y_b^2)}} \end{cases} \tag{4}\label{NA4}$$