I am asking here because I believe this is an issue with my understanding of mathematical notation, and not an issue with a physics misunderstanding although there is heavy physics context here.
I am writing a physics simulator, and right now I am working on simulating elastic collisions between spheres of equal mass (mass of 1 unit makes much of the math work out nicely).
I found this fantastic writeup https://physics.stackexchange.com/questions/107648/what-are-the-general-solutions-to-a-hard-sphere-collision which has gotten me most of the way there. If the spheres collide head on my simulation works but when they collide at an angle, the result shows energy was not conserved (energy lost). I feel I must be misunderstanding the notation.
The portion in question is the calculation of the impulse vector. Specifically this (copied from the link):
$$
J=2 \hat{j}\frac{\hat{j} \bullet R'}{\frac1{M_a} + \frac1{M_b}}
$$
Where R' is
$$
R'=V_{a1}-V_{b1}
$$
and j is the direction of the impulse (a unit vector through the COM of each sphere)
Considering that I have equal masses of 1 this equation should reduce to:
$$
J = \hat{j}\left( {\hat{j} \bullet R'}\right)
$$
I assume the quantity is a dot product of R' and the unit vector j.
If I have two spheres traveling toward each other such that they will collide at a 45 degree angle there is clearly something wrong here. If spheres A / B have velocities (say A is centered on the X-axis and B starts a bit above it, such that they collide at an angle)
$$
V_a = \begin{bmatrix} 5 \\ 0 \\ 0 \end{bmatrix}
V_b = \begin{bmatrix} -5 \\ 0 \\ 0 \end{bmatrix}
$$
My calculation for the direction of impulse is fine but I find
$$
\hat{j} = \begin{bmatrix} -\frac{1}{\sqrt{2}} \\ -\frac{1}{\sqrt{2}} \\ 0 \end{bmatrix}
$$
$$
R' = \begin{bmatrix} 10 \\ 0 \\ 0 \end{bmatrix}
$$
Which (here must be my misunderstanding) would mean
$$
J = \begin{bmatrix} -\frac{10}{\sqrt{2}} \\ 0 \\ 0 \end{bmatrix}
$$
This is clearly wrong, there needs to be a vertical component to the impulse vector. I assume the write-up is correct and I'm misunderstanding the notation, what is the correct way to interpret this?
edit:
The correct vector for reference is
$$
J = \begin{bmatrix} -5 \\ -5 \\ 0 \end{bmatrix}
$$
Which causes the spheres to change direction by 90 deg and continue on as expected.
Final edit for posterity:
In my final implementation I actually had to use the absolute value of the dot product. If it's negative, it flips the direction of the impulse and yields incorrect results. I am not sure if this is a mistake in the formula, or an artifact of my implementation. Either way it is working correctly now.
Best Answer
If $\hat{j}=\begin{bmatrix}-\frac{1}{\sqrt{2}}\\-\frac{1}{\sqrt{2}}\\0\end{bmatrix}$ and $R'=\begin{bmatrix}10\\0\\0\end{bmatrix}$, then
$$\hat{j}\cdot R'=-\frac{1}{\sqrt{2}}\times 10-\frac{1}{\sqrt{2}}\times 0+0\times 0=-\frac{10}{\sqrt{2}}$$
and so:
$$\hat{j}(\hat{j}\cdot R')=-\frac{10}{\sqrt{2}}\begin{bmatrix}-\frac{1}{\sqrt{2}}\\-\frac{1}{\sqrt{2}}\\0\end{bmatrix}=\begin{bmatrix}5\\5\\0\end{bmatrix}$$
Note that $\hat{j}\cdot R'$ is a scalar $-\frac{10}{\sqrt{2}}$, not a vector.