Hilbert spaces require a positive definite inner product and are very well studied. Have vector spaces with a real "inner product" been studied at all, and if so under what name? In other words $\langle x|y\rangle $ is a real number, not just positive definite, and $\langle x|x\rangle = 0$ does not imply $|x\rangle = 0$. A search of the internet thus far has turned up nothing.
Motivation: The study of hilbert spaces grew out of the concepts of vectors in 3D euclidean space with a positive definite inner product of those vectors. It is well known today that spacetime is non-euclidean and at least 4-dimensional with a real metric, not a positive definite one like euclidean spaces. There are some results for Riemannian manifolds with "pseudo-riemannian", non-positive definite metrics but so far I can't find any studies of vector spaces with such a property for the "inner product" on that space.
Clearly the $\langle x|x\rangle = 0$ for non-zero $|x\rangle$ would be the toughest axiom to deal with. Perhaps it is so tough that no one has tackled it or been able to come up with any useful results. It would seem though that the clear importance of real metrics in the physical universe would make it an extremely important area to investigate.
Best Answer
There are indefinite scalar product spaces. I suggest reading "Indefinite Linear Algebra and Applications" by Gohberg, Lancaster, Rodman. Applications are wide; to name a few: theory of relativity and the research of polarized light (mostly Minkowski space is used here), and matrix polynomials (nicely covered in "Matrix polynomials", again by Gohberg, Lancaster, Rodman).
Each indefinite scalar product space is induced by a nonsingular Hermitian indefinite matrix $J$ by $$[x, y] := \langle Jx, y \rangle = y^* J x,$$ although I've seen other variations as is usuall with this (for example, $x^* J y$).
The vectors which you've mentioned, $x \ne 0$, but $[x,x] = 0$, are usually called degenerate, although other names are used as well, for example neutral. In later terminology (I think used mostly by physicists), $x$ for which $[x,x] < 0$ is called negative, and if $[x,x] > 0$, $x$ is called positive.
The most common indefinite scalar product space is hypperbolic, induced by $J = \mathop{\rm diag}(j_1,\dots,j_n)$, where $j_k \in \{-1,1\}$. Minkowski space is usually defined by $j_1 = \pm 1$ and $j_k = \mp 1$ for $k > 1$, or $j_k = \pm 1$ for $k < n$ and $j_n = \mp 1$.
There are wider classes of scalar products on finite real and complex spaces (I think there was even some work on spaces of quaternions) than indefinite ones. For example, orthosymmetric products, for which $J$ need not be Hermitian, but $J^* = \tau J$ for some $\tau \in \mathbb{C}$ such that $|\tau| = 1$.
Another widely researched class are symplectic scalar products, induced by $J = \left[\begin{smallmatrix} & {\rm I}_n \\ -{\rm I}_n \end{smallmatrix}\right]$.
I suggest reading Tisseur, especially her "Structured Factorizations in Scalar Product Spaces, Higham, Sanja Singer (especially "Orthosymmetric block reflectors" with Saša Singer), Mehrmann, Mehl, maybe few of my own papers,...