correlationdata preprocessingmachine learningpca
One of the steps in performing whitening is decorrelation. I understand that decorrelation reduces the correlation among various input features, but haven't found a compelling reason why reduced correlation helps machine learning models perform better most of the times.
Is there any mathematical or intuitive explanation to explain this or it's mostly an empirical finding?
If we want to to look at lags over a long time in the past (or features derived from them like exponential moving averages or interactions between them) then there would be a large number of feature candidates. As you correctly mentioned a grid search would be expensive, even if you want to train a simple linear regression model. One approach that can be very helpful in this case would be to use a fast sub optimal feature selection method. For example you can use Greedy backward subset selection, Greedy backward/forward, or Lasso feature selection. This can be much faster and you can potentially look at much larger number of features. Based on my personal experience and also according to this paper if features are high correlated greedy backward/forward is outperforming Lasso: http://papers.nips.cc/paper/3586-adaptive-forward-backward-greedy-algorithm-for-sparse-learning-with-linear-models.pdf
Another intuition that can be helpful in many time series is that as we look more into the past the exact time becomes less important. For example in your example, the impact of rain fall on the flow of water, the rain fall on today and yesterday will probably have different coefficients in your model. But the rain fall on 365 days ago and on 366 days ago will probably have the same impact on the flow today. This facilitates application of transforms / feature engineering techniques that aggregates the data based on time. For example you can have a grid of exponential moving averages (AR(1) systems or IIR(1) filters in signal processing terms) as new features to model long term memory, followed by a linear combination of lagged data (FIR filters) to model short term memory. Note that you don't have to include all the generated features in your model and it is a good idea to perform feature selection to select a few of AR(1) systems and lags. I used a scheme similar to what I described above to extract features from multiple time series as shown in the following diagram.
Another techniques that is commonly used is to perform an unsupervised feature extraction method on the time series data. For example you can use PCA and keep only the most dominant principal components, or you can use discrete (Cosine) Fourier transform and only keep the strongest components. There are other transforms like Haar wavelet that can be useful in certain domains to extract features from time series.
Whitening is par for the course in computer vision applications, and may help a variety of machine learning algorithms converge to an optimal solution, beyond SVMs. (More on that towards the end of my answer.)
Now, we whiten data and get a round blob as output.
To put that more mathematically, whitening transforms a distribution using its eigenvectors $\boldsymbol{u}_j$ in such a way that its covariance matrix becomes the unit matrix. Bishop (1995) pp. 300 illustrates this in a stylised manner:
Whitening is a useful preprocessing step because it both decorrelates and normalises the inputs.
Decorrelation
The training step of machine learning algorithms is simply an optimisation problem, however it is defined. Whitening gives nice optimisation properties to the input variables, causing such optimisation steps to converge faster. The mechanism for this improvement is that it affects the condition number of the Hessian in steepest descent-style optimisation algorithms. Here are some sources for further reading:
Normalisation
The fact that input variables now have unit variance is an example of feature normalisation, which is a prerequisite for many ML algorithms. Indeed, SVMs (along with regularized linear regression and neural networks) requires that features be normalised to work effectively, so whitening may be improving your SVMs' performance significantly thanks only to the feature normalisation effect.
Best Answer
Let's understand why correlated variables are removed when fitting ML models.
If 2 or more variables are perfectly correlated with each other, then it makes sense to remove them and let only one of them stay, as all of them span/explain the same dimension, thus, not adding any additional information.
So, when your data is considerably large, you might want to decorrelate the dataset before fitting an ML model, so that the size can be trimmed which would result in the model training and performing faster due to less number of columns.
Also, some parametric models carry the risk of being erratic due to their covariance matrix becoming singular (Reason: Due to multiple columns spanning the same dimension).