You're correct. The standard basis for $P_2(R)$ is $\beta = \{1,x,x^2\}$, as you said, and the author is starting with $\beta$ and following the Gram-Schmidt algorithm to generate an orthogonal basis.
In reply to your comment:
Let me answer your questions about the inner product first. The author has introduced an important type of inner product here. Since the elements of $P_2(R)$ are integrable functions, we can define the inner product of two vectors $f(x),g(x) \in P_2(R)$ as
$$
\langle f(x),g(x) \rangle = \int_{-1}^1 f(t)g(t) \,dt.
$$
So for example, the inner product of $v = 1-3x + 2x^2$ and $w = 5 + 8x$ is
\begin{align}
\langle 1-3x + 2x^2, 5 + 8x \rangle &= \int_{-1}^1 (1 - 3t + 2t^2)(5 + 8t) \,dt \\
&= \int_{-1}^1 5+t+2 t^2+160 t^3 \,dt = \frac{2}{3}
\end{align}
I'll leave it up to you to verify that this is actually an inner product (worth checking, if you haven't already).
(Also, in case you were wondering: the choice of bounds $\pm 1$ doesn't really matter here; if you choose any $a$ and $b$ with $-\infty < a < b < \infty$ you get a new inner product $\langle f(x), g(x)\rangle_{a,b} = \int_a^b f(t)g(t) dt$ on $P_2(R)$!)
So to answer one of your questions, yes, you just substitute $1$ for both $f(x)$ and $g(x)$ when computing $\|v_1\|$ ($=\|1\|$):
$$
\|v_1\| = \langle v_1,v_1\rangle^{1/2} = \langle1,1\rangle^{1/2}
= \left(\int_{-1}^1 1 \cdot 1 \,dt\right)^{1/2}.
$$
Summary:
Following the Gram-Schmidt process to generate an orthogonal basis $\{v_1,v_2,v_3\}$ from the ordered basis $\beta = \{1,x,x^2\}$, we obtain the following vectors:
\begin{align}
v_1 &= 1 \text{ (since $1$ = the first vector in $\beta$)} \\
v_2 &= x - \text{proj}_{v_1}(x) = x - \text{proj}_{1}(x) \\
&= x - \frac{\langle x, 1 \rangle}{\langle 1, 1 \rangle} \cdot 1 = x - \frac{0}{2} \cdot 1 = x \\
v_3 &= x^2 - \text{proj}_{v_1}(x^2) - \text{proj}_{v_2}(x^2) \\
&= x^2 - \frac{
\langle x^2, 1 \rangle
}{
\langle 1, 1 \rangle
} \cdot 1
- \frac{
\langle x^2, x \rangle
}{
\langle x, x \rangle
} \cdot x \\
&= x^2 - \frac{
\int_{-1}^1 t^2 \cdot 1 \,dt
}{
\int_{-1}^1 1 \cdot 1 \,dt
} \cdot 1
- \frac{
\int_{-1}^1 t^2 \cdot t \,dt
}{
\int_{-1}^1 t \cdot t \,dt
} \cdot x
= x^2 - \frac{2/3}{2} \cdot 1 - \frac{0}{2/3} \cdot x = x^2 - \frac{1}{3}.
\end{align}
In other words, our new orthogonal basis is
$$
B = \{v_1,v_2,v_3\}.
$$
If you haven't before, it's worth thinking through why this gives you an orthogonal basis (and validating your conceptual understanding by confirming that $\langle v_i, v_j \rangle = 0$ for $i \neq j$.
We've gotten through the labor-intensive part of Gram-Schmidt.
Since $B$ is an orthogonal basis, we can arrive at an orthonormal basis $\{u_1, u_2, u_3\}$ by simply dividing each of the vectors $v_i$ by its norm $\|v_i\| = \sqrt{\langle v_i, v_i \rangle}$.
This is easily done after having calculated $B$, since you can reduce calculating $\|v_i\|$ into taking the square root of a sum of mostly known inner product values.
The resulting orthonormal basis $\{u_1, u_2, u_3\}$ is
\begin{align*}
u_1 &= v_1/\|v_1\| = 1/\|1\| = 1/\sqrt{2} \\
u_2 &= v_2/\|v_2\| = x/\|x\| = x/\left(\int_{-1}^1 t^2 \,dt\right)^{1/2} = \frac{1}{\sqrt{2/3}}x = \sqrt{\frac{3}{2}}x \\
u_3 &= v_3/\|v_3\| = \frac{x^2 - 1/3}{\left(\int_{-1}^1 (t^2 - 1/3)^2 \,dt\right)^{1/2}} = \frac{3\sqrt{5}}{2\sqrt{2}} \left(x^2 - \frac{1}{3}\right) = \frac{3\sqrt{5}}{2\sqrt{2}} x^2 - \frac{\sqrt{5}}{2\sqrt{2}}
\end{align*}
Edit: As @lojle and @RayBern both noticed, I accidentally omitted a projection term when calculating $v_3$. I've corrected the error and revised the surrounding commentary accordingly.
Hint: The norm associated with an inner product is defined by
$$
\|f\|^2=\langle f,f\rangle.
$$
So, for instance,
$$
\|1+x\|^2=\langle 1+x,1+x\rangle=\int_0^1(1+x)(1+x)\,dx=\frac{7}{3},
$$
and so $\|1+x\|=\sqrt{\frac{7}{3}}$.
So, to perform Gram-Schmidt: say we are given $f_1(x)=3$, $f_2(x)=1+x$, and $f_3(x)=x^2$, and we want to come up with an orthonormal basis $\{e_1,e_2,e_3\}$.
We start by taking $v_1=f_1=3$; then $e_1=\frac{v_1}{\|v_1\|}$. But
$$
\|v_1\|^2=\langle v_1,v_1\rangle=\int_0^1 v_1\cdot v_1\,dx=\int_0^19\,dx=9,
$$
and so $\| v_1\|=3$; hence $e_1=\frac{3}{3}=1$.
Now, we take $v_2:=f_2-\text{proj}_{e_1}(f_2)$, where $\text{proj}_a(b)$ is the projection of $b$ on to $a$. We have
$$
\text{proj}_{1}(1+x)=\frac{\langle 1,1+x\rangle}{\|e_1\|^2}\cdot 1=\frac{\int_0^1 1\cdot(1+x)\,dx}{1^2}\cdot1=\frac{3}{2},
$$
so that $v_2=(1+x)-\frac{3}{2}=-\frac{1}{2}+x$. If you check, you will find that $v_2$ is orthogonal to $e_1$, as it should be! So, we make it a unit vector by dividing by its norm. Here,
$$
\left\|-\frac{1}{2}+x\right\|^2=\left\langle-\frac{1}{2}+x,-\frac{1}{2}+x\right\rangle=\int_0^1\left(-\frac{1}{2}+x\right)^2\,dx=\frac{1}{12},
$$
so that $\|-\frac{1}{2}+x\|=\frac{1}{2\sqrt{3}}$, and we define
$$
e_2=\frac{v_2}{\|v_2\|}=-\sqrt{3}+2\sqrt{3}x.
$$
See if you can get the last one from there.
Best Answer
The problem lies in your definition of $\lvert v_2\rangle$. It should be$$\lvert v_2 \rangle = \frac{\lvert V_2'\rangle - \langle V_2'\rvert v_1 \rangle \lvert v_1 \rangle}{\sqrt{\langle V_2' \rvert V_2' \rangle}}.$$Using this definition, you will get that, indeed, $\langle v_1|v_2\rangle=0$. Unless your inner product is linear in the first variable and anti-linear in the second one. Then it should be$$\lvert v_2 \rangle = \frac{\lvert V_2'\rangle - \langle V_2'\rvert v_1 \rangle^*\lvert v_1 \rangle}{\sqrt{\langle V_2' \rvert V_2' \rangle}}.$$