You're supposed to curl your hand like so:
$\hskip 1.3in$
Your knuckles are supposed to form an angle while your fingers are otherwise straight, and your knuckles themselves point in the direction of $a$ while your fingertips point in the direction of $b$.
Using this, part (b) presumably wants you to use RHR to figure out which of the 8 octants $\vec{n}$ lies in (or between!), which would determine the sign of the components. I can't say much without seeing the picture of $a$ and $b$ though...
Update (sorry, MSPaint...):
$\hskip 1in$
I assume $a$ is on the $xy$ plane, so your thumb is too. This implies $z=0$. Moreover, the thumb is still close enough to the positively oriented $x$-ray (haha, pun) that it has positive $x$ component, but far enough from the positively oriented $y$-ray to have negative $y$ component.
To start with, you have transcribed the rule incorrectly.
In your source, the formula is given this way:
if $\mathbf a = \langle a_1,a_2,a_3\rangle$
and $\mathbf b = \langle b_1,b_2,b_3\rangle$,
their cross product is denoted $\mathbf a\times \mathbf b$
and defined analytically as the vector
$\langle a_2b_3 − a_3b_2, a_3b_1 − a_1b_3, a_1b_2 − a_2b_1\rangle.$
Transcribing this into your notation, that is,
$\mathbf a \to v,$ $\mathbf b \to u,$
$a_1 \to a_2,$ $a_2 \to b_2,$ $a_3 \to c_2,$
$b_1 \to a_1,$ $b_2 \to b_1,$ and $b_3 \to c_1$:
If $v=(a_2, b_2, c_2)$ and $u=(a_1, b_1, c_1),$ then
$$
v \times u
= \begin{bmatrix} b_2 c_1 - c_2 b_1 \\
c_2 a_1 - a_2 c_1 \\
a_2 b_1 - b_2 a_1 \end{bmatrix} .
$$
You set the second coordinate to $a_2 c_1 - c_2 a_1,$ opposite the correct sign.
On the other hand I'm not sure you actually meant to write $v \times u$;
if you change the left-hand side of your equation to $u \times v$ then the right-hand side you wrote will be correct as it is.
The next thing that is problematic in your analysis is that if you look "down" at the $xz$ plane from a place far along the positive $y$ axis, the positive $z$ axis should be on the right and the positive $x$ axis on the left, that is, what you see should be this:
The convention I would expect to follow is that angles are measured counterclockwise from the $z$ axis in this view, as the figure above illustrates for the vector $u.$
By saying that $c_1 = \lvert u\vert \sin \alpha,$
in effect you have decided to measure your angles clockwise from the positive $x$ axis. Reversing rotation direction from counterclockwise to clockwise is equivalent to flipping the direction of the normal vector.
Note that
$\langle 1,0,0\rangle \times \langle 0,0,1\rangle = \langle 0,-1,0\rangle,$
that is, $\hat x \times \hat z = -\hat y.$
There are so many inadvertent sign changes in your analysis that I find it hard to count them. But there are apparently an odd number of them, because they don't all cancel out to give you the expected answer.
Now for a personal opinion about the linked article,
https://www.researchgate.net/publication/305759205_An_Even_Simpler_Proof_of_the_Right-Hand_Rule.
I find this article unsatisfying.
Setting aside the failure to give any guidance about handling two vectors in the $x,z$ plane, where is the argument that projecting both vectors onto a coordinate plane will preserve the right-hand rule?
Such an argument possibly could be made, but it would make this "proof" not nearly as simple as it claims to be.
If you want a proof that is as simple as the one claimed in the linked paper,
I think you will have to accept as given that the cross-product formula is correct up to possible reversals of the signs of the three resulting coordinates.
That is, we assume that the magnitude of each coordinate of the cross-product is correct in the given formula, and we merely need to show that the sign is correct for the right-hand rule rather than for the left-hand rule.
In order to show this for the $y$ axis, it is sufficient to take any two vectors such that these two vectors and the $y$ axis do not all lie in one plane and show that the resulting $y$ coordinate has the correct sign.
In particular, we can take the vectors $\hat x = \langle 1,0,0\rangle$
and $\hat z = \langle 0,0,1\rangle.$
The right-hand rule requires that $\hat z \times \hat x = \hat y.$
We see that the formula satisfies this requirement, so it is correct for the $y$ component of the cross product.
Best Answer
Remember that $a$ points in direction $(3,4)$ which is little above $y=x$ ($45^{\circ}$) line.
Open your palm so your fingers point in direction of a. Now curl your four fingers towards b. Your stretched out thumb will point in direction of $a \times b$.
Easy to see $x$-component is positive and $y$-component negative. More precisely, the components will be proportional to $(4,-3)$ so that $$(3,4)\cdot(4,-3)=0$$