Start by finding three vectors, each of which is orthogonal to two of the given basis vectors and then try and find a matrix $A$ which transforms each basis vector into the vector you've found orthogonal to the other two. This matrix gives you the inner product.
I would first work out the matrix representation $A'$ of the inner product with respect to the given basis, because then the columns of $A'$ are just the vectors that you've already found (since you want to transform each basis element into each of these vectors). It is important that you express the vectors you've found in terms of the basis given, and not the standard basis for this part.
Once you've done this, we can relate $A$ and $A'$ by $A = P^{\dagger}A'P$ where $P$ is the change of basis matrix from the given basis to the standard basis.
The standard method of orthogonalisation works like this:
1) take a basis $\{a_n\}$
2) take first vector $a_1$ , normalise it, call it the first vector in the new basis: $$b_1:=\frac{a_1}{\sqrt{\langle a_1,a_1\rangle}}.$$
3) take second vector $a_2$, find its component orthogonal to the first vector in the new basis: $$a'_2=a_2- \langle a_2,b_1\rangle b_1 ,$$ normalise it, then call it the second vector in the new basis:
$$b_2:=\frac{a'_2}{\sqrt{\langle a'_2,a'_2\rangle}}.$$
4) repeat for all consecutive vectors - orthogonalisation comes for all previous vectors in new basis: $$a_k'=a_k- \sum_{j=1}^{k-1} \langle a_k,b_j\rangle b_j,\quad b_k= \frac{a_k'}{\sqrt{\langle a_k',a_k'\rangle}}.$$
So, let's take a canonical basis in $P_2[x]$: $\{1,x,x^2\}$.
First vector is $1$. It's norm with respect to our inner product is $2\int_0^11\cdot 1dx =2$, hence the first vector in the new basis is $b_1=\frac{1}{\sqrt 2}$.
Second vector is $x$. The orthogonalisation yields $$x- \langle x,1/\sqrt 2\rangle 1/\sqrt 2= x-\frac{1}{\sqrt 2}\cdot2\int_0^1\frac{x}{\sqrt 2}dx =x-\frac 12.$$
Can you now normalise it (i.e. find the second vector in the new basis) and then find the third vector in the new basis?
Best Answer
As mentioned in the comments, given a basis $\beta = \{v_1, \ldots, v_n\}$ of an inner product space $V$, applying the Gram-Schmidt orthogonalization process will produce an orthogonal basis $\gamma = \{u_1, \ldots, u_n\}$ for $V$. We can compute the $u_i$ using the formula given in the link: $$ u_k = v_k - \sum_{j=1}^{k-1} \frac{\langle u_j, v_k \rangle}{\langle u_j, u_j \rangle} u_j \, . $$
So we begin with the standard basis $\beta = \{1, x, x^2\}$. We can take $u_1 = 1$, so to find $u_2$ we need to compute $$ u_2 = x - \frac{\langle 1, x \rangle}{\langle 1, 1 \rangle} 1 \, . $$ Now \begin{align*} \langle 1, 1 \rangle &= \int_0^1 1 \, \overline{1} \ dx = 1& &&\langle 1, x \rangle &= \int_0^1 1 \, \overline{x} \ dx = \left. \frac{x^2}{2} \right|_0^1 = \frac{1}{2} \end{align*} so $$ u_2 = x - \frac{\langle 1, x \rangle}{\langle 1, 1 \rangle} 1 = x - \frac{1/2}{1} 1 = x - \frac{1}{2} \, . $$ Can you use the formula to compute $u_3$? Once you've done that, don't forget to normalize your vectors so they all have length $1$.