Hint: The norm associated with an inner product is defined by
$$
\|f\|^2=\langle f,f\rangle.
$$
So, for instance,
$$
\|1+x\|^2=\langle 1+x,1+x\rangle=\int_0^1(1+x)(1+x)\,dx=\frac{7}{3},
$$
and so $\|1+x\|=\sqrt{\frac{7}{3}}$.
So, to perform Gram-Schmidt: say we are given $f_1(x)=3$, $f_2(x)=1+x$, and $f_3(x)=x^2$, and we want to come up with an orthonormal basis $\{e_1,e_2,e_3\}$.
We start by taking $v_1=f_1=3$; then $e_1=\frac{v_1}{\|v_1\|}$. But
$$
\|v_1\|^2=\langle v_1,v_1\rangle=\int_0^1 v_1\cdot v_1\,dx=\int_0^19\,dx=9,
$$
and so $\| v_1\|=3$; hence $e_1=\frac{3}{3}=1$.
Now, we take $v_2:=f_2-\text{proj}_{e_1}(f_2)$, where $\text{proj}_a(b)$ is the projection of $b$ on to $a$. We have
$$
\text{proj}_{1}(1+x)=\frac{\langle 1,1+x\rangle}{\|e_1\|^2}\cdot 1=\frac{\int_0^1 1\cdot(1+x)\,dx}{1^2}\cdot1=\frac{3}{2},
$$
so that $v_2=(1+x)-\frac{3}{2}=-\frac{1}{2}+x$. If you check, you will find that $v_2$ is orthogonal to $e_1$, as it should be! So, we make it a unit vector by dividing by its norm. Here,
$$
\left\|-\frac{1}{2}+x\right\|^2=\left\langle-\frac{1}{2}+x,-\frac{1}{2}+x\right\rangle=\int_0^1\left(-\frac{1}{2}+x\right)^2\,dx=\frac{1}{12},
$$
so that $\|-\frac{1}{2}+x\|=\frac{1}{2\sqrt{3}}$, and we define
$$
e_2=\frac{v_2}{\|v_2\|}=-\sqrt{3}+2\sqrt{3}x.
$$
See if you can get the last one from there.
The plane is $H = \{ x | \langle d, x \rangle = \alpha \}$, where $d=(1,1,-1)^T$, and $\alpha = -1$. Let $v = (1,1,2)^T$.
The nearest point on $H$ to $v$ can be found by solving $\langle d, v+\lambda d \rangle = \alpha$, which gives $\lambda = \frac{\alpha-\langle d, v \rangle}{\|d\|^2}$.
The shortest distance is given by $\|v-(v+\lambda d) \| = |\lambda| \|d \| = | \frac{\alpha-\langle d, v \rangle}{\|d\|} | = \frac{1}{\sqrt{3}}$.
Addendum: To see why this is basically the Gram Schmidt process (or part thereof):
Let $b_2,b_3$ be any two vectors such that $d,b_2,b_3$ span $\mathbb{R}^3$. Apply the Gram Schmidt process to get $n_1,n_2,n_3$. We have $n_1 = \frac{d}{\|d\|} $, of course, and since $n_2 \bot d$ and $n_3 \bot d$, we see that $n_2,n_3$ lie on the plane $\langle d , x \rangle = 0$. Furthermore, we see that the point $\frac{\alpha}{\|d\|} n_1$ lies on the plane $H$ above. Hence every point $p$ on the plane is give by $p=\frac{\alpha}{\|d\|} n_1 + \beta_2 n_2 + \beta_3 n_3$, where $\beta_i$ are arbitrary.
Now project $v$ onto $n_1,n_2,n_3$ to get $v = \gamma_1 n_1 + \gamma_2 n_2 + \gamma_3 n_3$. Then the distance (or rather distance squared) from $v$ to a point on the plane is given by $\|v-p\|^2 = |\gamma_1-\frac{\alpha}{\|d\|}|^2+ |\gamma_2-\beta_2|^2 + |\gamma_3-\beta_2|^2$, from which we see that the minimum distance (choosing $\beta_2, \beta_3$ appropriately) is $|\gamma_1-\frac{\alpha}{\|d\|}|$. Since $\gamma_1 = \langle v, n_1 \rangle = \langle v, \frac{d}{\|d\|} \rangle$, we obtain the value $| \frac{\alpha-\langle d, v \rangle}{\|d\|} |$ as above.
Since we do not need to compute the nearest point, we only need compute $\gamma_1$, hence we only need the first step of the Gram Schmidt process applied to $d$ followed by the projection of $v$ onto this first element.
Best Answer
You're correct. The standard basis for $P_2(R)$ is $\beta = \{1,x,x^2\}$, as you said, and the author is starting with $\beta$ and following the Gram-Schmidt algorithm to generate an orthogonal basis.
In reply to your comment: Let me answer your questions about the inner product first. The author has introduced an important type of inner product here. Since the elements of $P_2(R)$ are integrable functions, we can define the inner product of two vectors $f(x),g(x) \in P_2(R)$ as $$ \langle f(x),g(x) \rangle = \int_{-1}^1 f(t)g(t) \,dt. $$ So for example, the inner product of $v = 1-3x + 2x^2$ and $w = 5 + 8x$ is \begin{align} \langle 1-3x + 2x^2, 5 + 8x \rangle &= \int_{-1}^1 (1 - 3t + 2t^2)(5 + 8t) \,dt \\ &= \int_{-1}^1 5+t+2 t^2+160 t^3 \,dt = \frac{2}{3} \end{align} I'll leave it up to you to verify that this is actually an inner product (worth checking, if you haven't already). (Also, in case you were wondering: the choice of bounds $\pm 1$ doesn't really matter here; if you choose any $a$ and $b$ with $-\infty < a < b < \infty$ you get a new inner product $\langle f(x), g(x)\rangle_{a,b} = \int_a^b f(t)g(t) dt$ on $P_2(R)$!)
So to answer one of your questions, yes, you just substitute $1$ for both $f(x)$ and $g(x)$ when computing $\|v_1\|$ ($=\|1\|$): $$ \|v_1\| = \langle v_1,v_1\rangle^{1/2} = \langle1,1\rangle^{1/2} = \left(\int_{-1}^1 1 \cdot 1 \,dt\right)^{1/2}. $$
Summary: Following the Gram-Schmidt process to generate an orthogonal basis $\{v_1,v_2,v_3\}$ from the ordered basis $\beta = \{1,x,x^2\}$, we obtain the following vectors: \begin{align} v_1 &= 1 \text{ (since $1$ = the first vector in $\beta$)} \\ v_2 &= x - \text{proj}_{v_1}(x) = x - \text{proj}_{1}(x) \\ &= x - \frac{\langle x, 1 \rangle}{\langle 1, 1 \rangle} \cdot 1 = x - \frac{0}{2} \cdot 1 = x \\ v_3 &= x^2 - \text{proj}_{v_1}(x^2) - \text{proj}_{v_2}(x^2) \\ &= x^2 - \frac{ \langle x^2, 1 \rangle }{ \langle 1, 1 \rangle } \cdot 1 - \frac{ \langle x^2, x \rangle }{ \langle x, x \rangle } \cdot x \\ &= x^2 - \frac{ \int_{-1}^1 t^2 \cdot 1 \,dt }{ \int_{-1}^1 1 \cdot 1 \,dt } \cdot 1 - \frac{ \int_{-1}^1 t^2 \cdot t \,dt }{ \int_{-1}^1 t \cdot t \,dt } \cdot x = x^2 - \frac{2/3}{2} \cdot 1 - \frac{0}{2/3} \cdot x = x^2 - \frac{1}{3}. \end{align} In other words, our new orthogonal basis is $$ B = \{v_1,v_2,v_3\}. $$ If you haven't before, it's worth thinking through why this gives you an orthogonal basis (and validating your conceptual understanding by confirming that $\langle v_i, v_j \rangle = 0$ for $i \neq j$.
We've gotten through the labor-intensive part of Gram-Schmidt. Since $B$ is an orthogonal basis, we can arrive at an orthonormal basis $\{u_1, u_2, u_3\}$ by simply dividing each of the vectors $v_i$ by its norm $\|v_i\| = \sqrt{\langle v_i, v_i \rangle}$. This is easily done after having calculated $B$, since you can reduce calculating $\|v_i\|$ into taking the square root of a sum of mostly known inner product values. The resulting orthonormal basis $\{u_1, u_2, u_3\}$ is \begin{align*} u_1 &= v_1/\|v_1\| = 1/\|1\| = 1/\sqrt{2} \\ u_2 &= v_2/\|v_2\| = x/\|x\| = x/\left(\int_{-1}^1 t^2 \,dt\right)^{1/2} = \frac{1}{\sqrt{2/3}}x = \sqrt{\frac{3}{2}}x \\ u_3 &= v_3/\|v_3\| = \frac{x^2 - 1/3}{\left(\int_{-1}^1 (t^2 - 1/3)^2 \,dt\right)^{1/2}} = \frac{3\sqrt{5}}{2\sqrt{2}} \left(x^2 - \frac{1}{3}\right) = \frac{3\sqrt{5}}{2\sqrt{2}} x^2 - \frac{\sqrt{5}}{2\sqrt{2}} \end{align*}
Edit: As @lojle and @RayBern both noticed, I accidentally omitted a projection term when calculating $v_3$. I've corrected the error and revised the surrounding commentary accordingly.