Since it looks to me like there's a mistake in the equation, I'll try to re-derive it. Hopefully that will make it a little clearer what's going on.
We'll start with Snell's law:
$$
\frac{\sin\theta_1}{\sin\theta_2}=\frac{n_2}{n_1}
$$
Now there are several vectors we're going to deal with, I'll go through them one by one.
- $\hat\imath$: The vector in the direction of the incoming ray.
- $\hat{t}$: The vector in the direction of the transmitted ray.
- $\hat{n}$: The vector perpendicular to the surface, the normal vector.
- $\hat{n}_\perp$: A vector perpendicular to the normal vector, in the plane of $\hat\imath$ and $\hat{t}$.
Let's write out some relations between these vectors. We can express $\hat\imath$ and $\hat{t}$ in terms of the direction of the surface and their angles to the surface.
$$
\hat\imath = -\cos\theta_1\hat{n}+\sin\theta_1\hat{n}_\perp \\
\hat{t} = -\cos\theta_2\hat{n}+\sin\theta_2\hat{n}_\perp
$$
(the minus sign comes from the fact that the ray is headed towards the surface.)
Since we don't know $\hat{n}_\perp$, let's eliminate it from those equations.
$$
\hat{n}_\perp = \csc\theta_1\hat\imath+\cot\theta_1\hat{n} \\
\hat{n}_\perp = \csc\theta_2\hat{t}+\cot\theta_2\hat{n} \\
\csc\theta_1\hat\imath+\cot\theta_1\hat{n} = \csc\theta_2\hat{t}+\cot\theta_2\hat{n}
$$
Let's try to solve for $\hat{t}$.
$$
\csc\theta_2\hat{t}=\csc\theta_1\hat\imath+\cot\theta_1\hat{n}-\cot\theta_2\hat{n} \\
\hat{t}=\frac{\sin\theta_2}{\sin\theta_1}\hat\imath+\cos\theta_1\frac{\sin\theta_2}{\sin\theta_1}\hat{n}-\cos\theta_2\hat{n}
$$
Those ratios of $\sin$'s are in Snell's law, so let's replace them and consolidate terms.
$$
\hat{t}=\frac{n_1}{n_2}\hat\imath+\left(\cos\theta_1\frac{n_1}{n_2}-\cos\theta_2\right)\hat{n}
$$
Finally let's replace the $\cos$ with a $\sin$.
$$
\hat{t}=\frac{n_1}{n_2}\hat\imath+\left(\cos\theta_1\frac{n_1}{n_2}-\sqrt{1-\sin^2\theta_2}\right)\hat{n}
$$
So we've recovered (almost) your original expression. It gives you the direction of the transmitted ray given the direction of the surface normal and incoming ray.
Note that $\cos\theta_1=\hat\imath\cdot\hat{n}$ and $\sin\theta_2=\frac{n_1}{n_2}\sqrt{1-\cos^2\theta_1}$.
In order to implement this equation, you'll need to add and multiply vectors, so let's talk vector arithmetic. For a more complete understanding I recommend you do some reading up on the subject.
A vector $\vec{v}$ has 3 components (in 3d space), I'll denote them $v_x$, $v_y$, and $v_z$.
The product of a vector $\vec{v}$ and a scalar (regular number) $a$ is a vector:
$$
a\times(v_x,v_y,v_z) = (av_x,av_y,av_z)
$$
The sum of two vectors $\vec{v}$ and $\vec{u}$ is a vector:
$$
(v_x,v_y,v_z)+(u_x,u_y,u_z) = (v_x+u_x,v_y+u_y,v_z+u_z)
$$
The dot product of two vectors $\vec{v}$ and $\vec{u}$ is a scalar:
$$
(v_x,v_y,v_z)\cdot(u_x,u_y,u_z) = v_x u_x+v_y u_y+v_z u_z
$$
The magnitude (length) of a vector $\vec{v}$ is $|\vec{v}|=\sqrt{\vec{v}\cdot\vec{v}}$. Directions, like the directions of the light rays in raytracing, are represented by unit vectors: vectors with unit magnitude. We typically denote unit vectors with a hat instead of an arrow (e.g. $\hat{n}$). For a unit vector $\hat{n}$, it's always true that $\hat{n}\cdot\hat{n}=1$. To make a unit vector (normalize) any vector, simply divide that vector by its magnitude (equivalent to multiplying by one over the magnitude).
Note that you'll sometimes see the three coordinate axes labeled by $i$, $j$, and $k$ or $1$, $2$, and $3$. Also, you'll see vectors written as sums of the unit vectors along the coordinate axes, e.g:
$$
\vec{v}=(v_x,v_y,v_z) \\
=v_x\hat{x}+v_y\hat{y}+v_z\hat{z} \\
=v_x\hat{i}+v_y\hat{j}+v_z\hat{k} \\
=v_x\hat{e}_1+v_y\hat{e}_2+v_z\hat{e}_3
$$
Best Answer
I've added the homework tag even though it likely isn't. I think the best way to make sure of equations like this is simply to derive them yourself - that way you won't miss any quirky or unwonted conventions the author may be using.
So here we have the basic facts:
So, translate these into equations: (1) says that the three vectors are linearly dependent or:
$$\vec{t} = \alpha\,\vec{i} + \beta\,\vec{n};\;\alpha,\,\beta\in \mathbb{R}$$
(3) says that $n_1\, \vec{i} \wedge \vec{n} = n_2\, \vec{t} \wedge \vec{n}$ which simplifies to $n_1 = n_2 \alpha$. Now all we need is $|\vec{t}|^2 = \vec{t}\cdot \vec{t} = \alpha^2 + \beta^2 - 2 \alpha \beta \cos\theta_1 = 1$ and we can solve these equations to give the right expressions.