The choice of inner product defines the notion of orthogonality.
The usual notion of being "perpendicular" depends on the notion of "angle" which turns out to depend on the notion of "dot product".
If you change the way we measure the "dot product" to give a more general inner product then we change what we mean by "angle", and so have a new notion of being "perpendicular", which in general we call orthogonality.
So when you apply the Gram-Schmidt procedure to these vectors you will NOT necessarily get vectors that are perpendicular in the usual sense (their dot product might not be $0$).
Let's apply the procedure.
It says that to get an orthogonal basis we start with one of the vectors, say $u_1 = (-1,1,0)$ as the first element of our new basis.
Then we do the following calculation to get the second vector in our new basis:
$u_2 = v_2 - \frac{\langle v_2, u_1\rangle}{\langle u_1, u_1\rangle} u_1$
where $v_2 = (-1,1,2)$.
Now $\langle v_2, u_1\rangle = 3$ and $\langle u_1, u_1\rangle = 3$ so that we are given:
$u_2 = v_2 - u_1 = (0,0,2)$.
So your basis is correct. Let's check that these vectors are indeed orthogonal. Remember, this is with respect to our new inner product. We find that:
$\langle u_1, u_2\rangle = 3(-1)(0) + (1)(0) + 2(0)(2) = 0$
(here we also happened to get a basis that is perpendicular in the traditional sense, this was lucky).
Now is the basis orthonormal? (in other words, are these unit vectors?). No they arent, so to get an orthonormal basis we must divide each by its length. Now this is not the length in the usual sense of the word, because yet again this is something that depends on the inner product you use. The usual Pythagorean way of finding the length of a vector is:
$||x||=\sqrt{x_1^2 + ... + x_n^2} = \sqrt{x . x}$
It is just the square root of the dot product with itself. So with more general inner products we can define a "length" via:
$||x|| = \sqrt{\langle x,x\rangle}$.
With this length we see that:
$||u_1|| = \sqrt{2(-1)(-1) + (1)(1) + 3(0)(0)} = \sqrt{3}$
$||u_2|| = \sqrt{2(0)(0) + (0)(0) + 3(2)(2)} = 2\sqrt{3}$
(notice how these are different to what you would usually get using the Pythagorean way).
Thus an orthonormal basis is given by:
$\{\frac{u_1}{||u_1||}, \frac{u_2}{||u_2||}\} = \{(\frac{-1}{\sqrt{3}}, \frac{1}{\sqrt{3}}, 0), (0,0,\frac{1}{\sqrt{3}})\}$
In follow,
with the star symbol, I mean complex conjugate, i.e. for example
$(3-2i,3,2i)^*=(3+2i,3,-2i)$,
If $v=\{{v_1,v_2,v_3}\}$ and $u=\{{u_1,u_2,u_3}\}$ , we define
$v.u=v_1u_1+v_2 u_2+v_3 u_3$ and $<v,u>=u^*.v$.
The detailed answer is as follow,
1.
$v_1=(1,i,0)$ and $v_2=(1-i,2,4i)$, using the Gram Schmidt process, we have
$$w_1=v_1=(1,i,0)$$
$$w_2=v_2-\frac{v_1^*.v_2}{v_1^*.v_1}v_1=(1-i,2,4i)-\frac{(1,-i,0).(1-i,2,4i)}{(1,-i,0).(1,i,0)}(1,i,0)\\ =(1-i,2,4i)-\frac{1-3i}{2}(1,i,0)=(\frac{1+i}{2},\frac{1-i}{2},4i)$$
2.
$$u_1=\frac{{w}_1}{\sqrt{{w}_1^*.{w}_1}}=\frac{(1,i,0)}{\sqrt{(1,-i,0).(1,i,0)}}=\frac{1}{\sqrt{2}}(1,i,0)$$
$$u_2=\frac{{w}_2}{\sqrt{{w}_2^*.{w}_2}}=\frac{(\frac{1+i}{2},\frac{1-i}{2},4i)}{\sqrt{(\frac{1-i}{2},\frac{1+i}{2},-4i).(\frac{1+i}{2},\frac{1-i}{2},4i)}}\\=\frac{1}{\sqrt{17}}(\frac{1+i}{2},\frac{1-i}{2},4i)=\frac{1}{2\sqrt{17}}(1+i,1-i,8i)$$
3.
Since $x \in {C^3}$, we need three orthonormal basis $\{{u_1,u_2,u_3}\}$. The unit vector $u_3$ must be orthogonal to both $u_1$ and $u_2$, therefore, if we assume $u_3=(\alpha_1,\alpha_2,\alpha_3)$ then,
$u_1^*.u_3=0 \to \frac{1}{\sqrt{2}}(1,-i,0).(\alpha_1,\alpha_2,\alpha_3)=0 \to \alpha_1-i\alpha_2=0 \,\,(1)$
$u_2^*.u_3=0 \to \frac{1}{2\sqrt{17}}(1-i,1+i,-8i).(\alpha_1,\alpha_2,\alpha_3)=0 \to (1-i)\alpha_1+(1+i)\alpha_2-8i\alpha_3=0 \,\,(2)$
Equations $(1)$ and $(2)$ have solutions of the form of $\alpha (4i,4,1 - i)$. Using the normalization condition,
$$\sqrt {{{\left| {{\alpha _1}} \right|}^2} + {{\left| {{\alpha _2}} \right|}^2} + {{\left| {{\alpha _3}} \right|}^2}} = 1$$
we obtain,
$$u_3=\frac{1}{\sqrt{34}}(4i,4,1-i)$$
Now, we turn into calculation of Fourier coefficients. From vector equation $$a_1u_1+a_2u_2+a_3u_3=x$$
or
$$\frac{a_1}{\sqrt{2}}(1,i,0)+\frac{a_2}{2\sqrt{17}}(1+i,1-i,8i)+\frac{a_3}{\sqrt{34}}(4i,4,1-i)=(3+i,4i,-4)$$
we have these linear system of equations,
$$\frac{a_1}{\sqrt{2}}+\frac{a_2}{2\sqrt{17}}(1+i)+\frac{4ia_3}{\sqrt{34}}=3+i$$
$$\frac{i a_1}{\sqrt{2}}+\frac{a_2}{2\sqrt{17}}(1-i)+\frac{4a_3}{\sqrt{34}}=4i$$
$$\frac{8ia_2}{2\sqrt{17}}+\frac{a_3}{\sqrt{34}}(1-i)=-4$$
which can be solved easily,
$$a_1=\frac{7+i}{\sqrt{2}}$$
$$a_2=\sqrt{17}i$$
$$a_3=0$$
4.
Using given equation, we have
$$a_1=u_1^*.x=\frac{1}{\sqrt{2}}(1,-i,0).(3+i,4i,-4)=\frac{7+i}{\sqrt{2}}$$
$$a_2=u_2^*.x=\frac{1}{2\sqrt{17}}(1-i,1+i,-8i).(3+i,4i,-4)=\sqrt{17}i$$
$$a_3=u_3^*.x=\frac{1}{\sqrt{34}}(-4i,4,1+i).(3+i,4i,-4)=0$$
Best Answer
Asserting that $(u_1,u_2,u_3)$ is an orthonormal basis of that space consists in checking several things:
Whover wrote this is not claming that just because $\lVert u_3\rVert=1$, then $(u_1,u_2,u_3)$ is an orthonormal basis. That was just the last thing that had to be checked.