Let $q: \mathbb{R^3}\to \mathbb{R},\ q(x_1, x_2, x_3)=-5x_1^2-x_2^2-x_3^2+2x_1x_3+2x_2x_3-4x_1x_2$ be a quadratic form.
Find a maximal subspace $W \subseteq \mathbb{R^3}$ such that $\forall w \in W,\ q(w) \geq 0$.
Using row/column operations on $[q]_e$, I got to the canonical form: $\text{diag}(-1, -1, 1)$.
That tells me that there exists a basis $(u)=(u_1, u_2, u_3)$ such that: $$[v]_u=\begin{pmatrix}
x \\
y \\
z
\end{pmatrix},\ q(v)=-x^2-y^2+z^2$$
So if $w=\alpha u_3,\ q(w)=\alpha^2 \geq 0$, therefore $w \in W$. So far I've shown $\text{span}(u_3) \subseteq W$.
If we assume that there exists $W\subset {W}'$ such that ${W}'$ is maximal, then $\forall {w}' \in {W}'$: $$q({w}')=-w_1^2-w_2^2+w_3^2\geq 0,\ [{w}']_u=\begin{pmatrix}
w_1 \\
w_2 \\
w_3
\end{pmatrix}$$
But if we assume ${w}' \notin W$ then $w_1 \neq 0$ or $w_2 \neq 0$.
At this point I'm stuck, not sure how to finish this up.
Best Answer
It is a standard result that the signature of a quadratic form is well-defined. You have found the canonical form $\text{diag}(-1,-1,1)$, so the signature is $(1,2)$. Now the number of +1's (the first number of the signature) equals the maximal dimension of a subspace on which the form is positive definite. E.g. see my proof here. So in this case any 1-dimensional space on which the form is positive definite, is maximal. So your $\text{span}(u_3)$ is indeed maximal.
Let me apply the argument I linked to to the current case. Suppose $V\supset \text{span}(u_3)$ is a subspace on which $q$ is positive definite. Then $q$ is both positive and negative definite on the subspace $V\cap \text{span}(u_1,u_2)$. Of course this implies that this subspace is $\{0\}$. But, $\{u_1,u_2,u_3\}$ being a basis, $V\supset \text{span}(u_3)$ combined with $V\cap \text{span}(u_1,u_2)=\{0\}$ implies $V= \text{span}(u_3)$. Hence $\text{span}(u_3)$ is maximal.