Added: the cleanest expression before reaching the $\pm 1$ stage is
$$ 8\left(2x - y - z \right)^2 - 3\left(3x - y - 2z \right)^2 = 5 x^2 + 5 y^2 - 4 z^2 + 4 yz + 4 zx - 14 xy.$$ This becomes
$$ \left( \sqrt {32} \;x - \sqrt 8 \; y - \sqrt 8 \; z \right)^2 - \left( \sqrt {27} \;x - \sqrt 3 \; y - \sqrt {12} \;z \right)^2 = 5 x^2 + 5 y^2 - 4 z^2 + 4 yz + 4 zx - 14 xy.$$
$$ P^T H P = D $$
$$\left(
\begin{array}{rrr}
1 & 0 & 0 \\
\frac{ 7 }{ 5 } & 1 & 0 \\
1 & 1 & 1 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
5 & - 7 & 2 \\
- 7 & 5 & 2 \\
2 & 2 & - 4 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
1 & \frac{ 7 }{ 5 } & 1 \\
0 & 1 & 1 \\
0 & 0 & 1 \\
\end{array}
\right)
= \left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - \frac{ 24 }{ 5 } & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
$$
$$ Q^T D Q = H $$
$$\left(
\begin{array}{rrr}
1 & 0 & 0 \\
- \frac{ 7 }{ 5 } & 1 & 0 \\
\frac{ 2 }{ 5 } & - 1 & 1 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - \frac{ 24 }{ 5 } & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
1 & - \frac{ 7 }{ 5 } & \frac{ 2 }{ 5 } \\
0 & 1 & - 1 \\
0 & 0 & 1 \\
\end{array}
\right)
= \left(
\begin{array}{rrr}
5 & - 7 & 2 \\
- 7 & 5 & 2 \\
2 & 2 & - 4 \\
\end{array}
\right)
$$
At this stage, to force diagonal elements to be $\pm 1,$ we will need to introduce some square roots, but we can throw these in a diagonal matrix that is its own transpose $S^T = S, \; \; $ let us call it $S$ for square root,
$$
S =
\left(
\begin{array}{rrr}
\frac{1}{\sqrt 5} & 0 & 0 \\
0 & \frac{\sqrt 5}{ \sqrt{24} } & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
$$
so that
$$ S^T D S =
\left(
\begin{array}{rrr}
1 & 0 & 0 \\
0 & - 1 & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
$$
Finally
$$ S^T P^T H P S =
\left(
\begin{array}{rrr}
1 & 0 & 0 \\
0 & - 1 & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
$$
and your basis matrix is $R = PS$
The algorithm for the first part, all rational entries, is:
Algorithm discussed at http://math.stackexchange.com/questions/1388421/reference-for-linear-algebra-books-that-teach-reverse-hermite-method-for-symmetr
$$ H = \left(
\begin{array}{rrr}
5 & - 7 & 2 \\
- 7 & 5 & 2 \\
2 & 2 & - 4 \\
\end{array}
\right)
$$
$$ D_0 = H $$
$$ E_j^T D_{j-1} E_j = D_j $$
$$ P_{j-1} E_j = P_j $$
$$ E_j^{-1} Q_{j-1} = Q_j $$
$$ P_j Q_j = I $$
$$ P_j^T H P_j = D_j $$
$$ Q_j^T D_j Q_j = H $$
$$ H = \left(
\begin{array}{rrr}
5 & - 7 & 2 \\
- 7 & 5 & 2 \\
2 & 2 & - 4 \\
\end{array}
\right)
$$
==============================================
$$ E_{1} = \left(
\begin{array}{rrr}
1 & \frac{ 7 }{ 5 } & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
$$
$$ P_{1} = \left(
\begin{array}{rrr}
1 & \frac{ 7 }{ 5 } & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; Q_{1} = \left(
\begin{array}{rrr}
1 & - \frac{ 7 }{ 5 } & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; D_{1} = \left(
\begin{array}{rrr}
5 & 0 & 2 \\
0 & - \frac{ 24 }{ 5 } & \frac{ 24 }{ 5 } \\
2 & \frac{ 24 }{ 5 } & - 4 \\
\end{array}
\right)
$$
==============================================
$$ E_{2} = \left(
\begin{array}{rrr}
1 & 0 & - \frac{ 2 }{ 5 } \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
$$
$$ P_{2} = \left(
\begin{array}{rrr}
1 & \frac{ 7 }{ 5 } & - \frac{ 2 }{ 5 } \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; Q_{2} = \left(
\begin{array}{rrr}
1 & - \frac{ 7 }{ 5 } & \frac{ 2 }{ 5 } \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; D_{2} = \left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - \frac{ 24 }{ 5 } & \frac{ 24 }{ 5 } \\
0 & \frac{ 24 }{ 5 } & - \frac{ 24 }{ 5 } \\
\end{array}
\right)
$$
==============================================
$$ E_{3} = \left(
\begin{array}{rrr}
1 & 0 & 0 \\
0 & 1 & 1 \\
0 & 0 & 1 \\
\end{array}
\right)
$$
$$ P_{3} = \left(
\begin{array}{rrr}
1 & \frac{ 7 }{ 5 } & 1 \\
0 & 1 & 1 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; Q_{3} = \left(
\begin{array}{rrr}
1 & - \frac{ 7 }{ 5 } & \frac{ 2 }{ 5 } \\
0 & 1 & - 1 \\
0 & 0 & 1 \\
\end{array}
\right)
, \; \; \; D_{3} = \left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - \frac{ 24 }{ 5 } & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
$$
==============================================
$$ P^T H P = D $$
$$\left(
\begin{array}{rrr}
1 & 0 & 0 \\
\frac{ 7 }{ 5 } & 1 & 0 \\
1 & 1 & 1 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
5 & - 7 & 2 \\
- 7 & 5 & 2 \\
2 & 2 & - 4 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
1 & \frac{ 7 }{ 5 } & 1 \\
0 & 1 & 1 \\
0 & 0 & 1 \\
\end{array}
\right)
= \left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - \frac{ 24 }{ 5 } & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
$$
$$ Q^T D Q = H $$
$$\left(
\begin{array}{rrr}
1 & 0 & 0 \\
- \frac{ 7 }{ 5 } & 1 & 0 \\
\frac{ 2 }{ 5 } & - 1 & 1 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
5 & 0 & 0 \\
0 & - \frac{ 24 }{ 5 } & 0 \\
0 & 0 & 0 \\
\end{array}
\right)
\left(
\begin{array}{rrr}
1 & - \frac{ 7 }{ 5 } & \frac{ 2 }{ 5 } \\
0 & 1 & - 1 \\
0 & 0 & 1 \\
\end{array}
\right)
= \left(
\begin{array}{rrr}
5 & - 7 & 2 \\
- 7 & 5 & 2 \\
2 & 2 & - 4 \\
\end{array}
\right)
$$
There can't be an orthonormal basis for which the quadratic form has canonical diagonal form, because if there were such a basis $\{v_1,v_2,v_3\}$, then for any unit vector $u = \sum_i \lambda_i v_i$ we would have $$q(u) = \left(\sum_i \lambda_i v_i\right)^T A \left(\sum_i \lambda_i v_i\right) = \begin{bmatrix}\lambda_1 & \lambda_2 & \lambda_3\end{bmatrix} \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 0
\end{bmatrix}
\begin{bmatrix}\lambda_1\\ \lambda_2 \\ \lambda_3\end{bmatrix} \leq \sum_i \lambda_i^2 = 1$$
But this is false, because from your definition of $q$, we see that e.g. $q(0,1,0)=8$.
By writing $A$ as a symmetric matrix you can find an orthonormal basis that diagnonalizes it (because a symmetric matrix always can always be diagonalized by a rotation matrix), but the diagonal entries will then be the eigenvalues of $A$, not $1$.
Best Answer
Even if $\{e_1,\dots,e_n\}$ is not an orthonormal basis, the formula $Q(x)$ defines a quadratic form on $V$. That said, when $A$ is described as being "the" matrix associated with a quadratic form, the basis relative to which the quadratic form is defined is assumed to be orthonormal.