I have heard and read the term energy in the context of LA a few times now. e.g:
The algorithm is based on the geometrical observations that the word
embeddings (across all representations such as Glove, word2vec etc.)
have a large mean vector and most of their energy (after subtracting the mean
vector) is located in a subspace of about 8 dimensions.
I am having a hard time finding explanation on this energy concept however. Searching for subspace energy will give you Star Trek related results for example 😀
Searching for linear algebra energy will result in things like What is the importance of linear algebra in renewable energy engineering?
So, can you explain to me what energy in LA is and how I can find material on it?
Best Answer
It can reasonably be inferred from your quote that the phrasing have most of one's energy in subspace $V$ is an empiric way to say that a vector "almost lies in" $V$.
There are different ways you can measure that. For instance, in a Euclidean space, the ratio $$\frac{\|\text{orthogonal projection of $v$ onto $V$}\|}{\|v\|}$$ somewhat tells you the "proportion" of $v$ that lies in $V$.
A practical framework where this could occur is signal analysis, where the energy of a signal is the sum of squares of its Fourier components.