Looking at different definitions and types of Entropy, I run into the concept of SVD Entropy, which is defined as explained below.
What is the intuition behind the SVD spectrum? What do different frequencies in it capture? And most importantly what does its Entropy measure, in relation to the original signal?
$\quad$*SVD Entropy*
$\quad$$[17]$ defines an entropy measure using Singular Value Decomposition (SVD). Let the input signal be $[x_1,x_2,\cdots,x_n]$. We construct delay vectors as: $$\mathbf y(i)=[x_i,x_{i+\tau},\cdots,x_{i+(d_E-1)\tau}],$$$\quad$where $\tau$ is the delay and $d_E$ is the embedding dimension. In this paper, $d_E=20$ and $\tau=2$. The embedding space is then constructed by: $$Y=[\mathbf y(1),\mathbf y(2),\cdots,\mathbf y(N-(d_E-1)\tau)]^T$$$\quad$The SVD is then performed on matrix $Y$ to produce $M$ singular values, $\sigma_1,\cdots,\sigma_M$, known as the singular spectrum.
$\quad$The SVD entropy is then defined as: $$H_{SVD}=-\sum_{i=1}^M\overline \sigma_i\log_2\overline\sigma_i\tag{4}$$ where $M$ is the number of singular values and $\sigma_1,\cdots,\sigma_M$ are normalized singular values by $\overline\sigma_i=\sigma_i/\sum_{j=1}^M\sigma_j$. SVD entropy is a scalar feature.
Best Answer
Since the question begins with the word "intuitive", here's one way to describe it intuitively.
The SVD weights indicate how much of the data set is explained by each vector.
Entropy delivers the maximally-noncommittal data set at a given signal-noise ratio, that is to say, the most information with the least artefact.
So the SVD entropy is an indicator of how many vectors are needed for an adequate explanation of the data set.
You could say it measures feature-richness in the sense that the higher the entropy of the set of SVD weights, the more orthogonal vectors are required to adequately explain it.