We have:
$$\langle g,h\rangle = 0,\quad \langle g,g\rangle = \frac{1}{12},\quad \langle h,h\rangle = 1,$$
while:
$$ \langle f,g \rangle = \frac{1}{3},\quad \langle f,h \rangle = -\frac{8}{3},$$
hence the projection of $f$ on $\operatorname{Span}(g,h)$ is given by:
$$ f^{\perp} =4g-\frac{8}{3}h = 4x-\frac{14}{3}. $$
I'm assuming the projection is from $V$ to $V$, so that we can enter in a $2 \times 2$ matrix.
First, write out the formula for orthogonal projection:
$$\mathrm{proj}_v(x) = \frac{v \cdot x}{v \cdot v} v$$
Next, transform the basis vectors:
$$\mathrm{proj}_v(-4, 1, 0) = \frac{(-12, 4, -2) \cdot (-4, 1, 0)}{(-12, 4, -2) \cdot (-12, 4, -2)} (-12, 4, -2) = \frac{13}{41} (-12, 4, -2)$$
You can do the other one. Then, write these transformed basis vectors as coordinates of the basis. We can compute the coordinates of the first transformed vector by row-reducing the augmented matrix (the third column is augmented):
$$\left(\begin{matrix}-4 & -2 & -\frac{156}{41} \\ 1 & 0 & \frac{52}{41} \\ 0 & 1 & -\frac{26}{41} \end{matrix}\right).$$
This row reduces to:
$$\left(\begin{matrix}1 & 0 & \frac{52}{41} \\ 0 & 1 & -\frac{26}{41} \\ 0 & 0 & 0 \end{matrix}\right).$$
Thus, we have:
$$\mathrm{proj}_v(-4, 1, 0) = \frac{13}{41} (-12, 4, -2) = \frac{52}{41}(-4, 1, 0) - \frac{26}{41}(-2, 0, 1).$$
Therefore, the first column vector of the matrix is:
$$\left(\begin{matrix}\frac{52}{41} \\ \frac{-26}{41}\end{matrix}\right).$$
Now do the same for the other vector, and you are done!
Best Answer
Hints:
Clearly $\;\{1\,,\,x-12\}\;$ is a basis for $\;U:=Span\{1\,,\,x-12\}\;$. Now, apply Gram-Schmidt to orthonormalize this basis:
$$||1||:=\left(\int\limits_a^bdx\right)^{1/2}=\sqrt{b-a}\implies u_1=\frac1{\sqrt{b-a}}$$
$$v_2=(x-12)-\frac{\langle x-12\,,\,u_1\rangle}{||u_1||^2}u_1=(x-12)-\frac1{b-a}\left(\int\limits_a^b (x-12)dx\right)=$$
$$=(x-12)-\frac1{b-a}\left(\frac12(b^2-a^2)-12(b-a)\right)=x-12-\frac12(a+b)+12=$$
$$=x-\frac12(a+b)\implies ||v_2||=\left(\int\limits_a^b\left(x-\frac12(a+b)\right)dx\right)^{1/2}=\ldots\implies u_2=\frac{v_2}{||v_2||}$$
The set $\;\{u_1,u_2\}\;$ is an orthonormal base of $\;U\;$, and thus the orthogonal projection on $\;U\;$ is given by
$$P_U:=u_1u_1^t+u_2u_2^t$$
Other way to do it: Using the hint you were given, prove that in fact
$$P_U=A\left(A^tA\right)^{-1}A^t\;,\;\;\text{with}\;\;A=\text{the matrix according to the original vectors}$$
$$1\,,\,x-12\;\;\text{of}\;\;U$$