[Math] Inner Product Space vs. Vector Space

inner-productslinear algebramachine learning

I had no trouble understanding what a vector space is: a constraint on the type of vectors you can create, such that certain operations could be performed with them.

For example, a vector space of
$$\left( \begin{array}{ccc}
a \\
a\\
a\end{array} \right)$$

fulfills all the requirements of vector spaces and is represented by a straight line through the origin.

However I cannot understand what an Inner Product Space is. My first thought was that it's a set of vectors that fulfill certain requirements like Symmetry, Positivity, Multiplicity etc, similar to the concept of vector space. However, upon further observation, I realized that all vector spaces have these properties. If it is a vector space, it seems it is already an Inner Product Space.

So what exactly does the Inner Product Space do for us? What does it add?
Is there a visual representation of an Inner Product Space?

Best Answer

An inner product is an additional structure on a vector space. It is true that all vector spaces have inner products, but there can be two different inner products on the same vector space. For instance, on $\mathbb{R}^2$ one has the "usual" inner product $\langle a,b \rangle \cdot \langle c,d \rangle= ac+bd$. But if you change coordinates by replacing the "standard basis" $\langle 1,0 \rangle$, $\langle 0,1 \rangle$ by some other basis then you may get other inner products.

The additional structure that an inner products gives to a vector space is geometric in nature. First, the inner product gives a way of measuring lengths of vectors, using the formula $$Length(v) = \sqrt{v \cdot v} $$ Second, the inner product gives a way of measuring angles between two vectors, using the law of cosines formula $$Angle(v,w) = \cos^{-1}[(v \cdot w) / (Length(v) \, Length(w))] $$

If you change inner products on the same vector space, then you may get two different angle measurements, two different length measurements, two different notions of "circles", etc. Just as an example, if you define an inner product on $\mathbb{R}^2$ using the basis $\langle 2,0 \rangle$, $\langle 0,1 \rangle$, then the "circles" in this geometry are ellipses whose major axis along the $x$-axis has twice the ordinary Euclidean length as their minor axis along the $y$-axis.